_ _ ____ _ ___| | | | _ \| | / __| | | | |_) | | | (__| |_| | _ <| |___ \___|\___/|_| \_\_____| Changelog Daniel (16 January 2006) - David Shaw finally removed all traces of Gopher and we are now officially not supporting it. It hasn't been functioning for years anyway, so this is just finally stating what already was true. And a cleanup at the same time. - Bryan Henderson turned the 'initialized' variable for curl_global_init() into a counter, and thus you can now do multiple curl_global_init() and you are then supposed to do the same amount of calls to curl_global_cleanup(). Bryan has also updated the docs accordingly. Daniel (13 January 2006) - Andrew Benham fixed a race condition in the test suite that could cause the test script to kill all processes in the current process group! Daniel (12 January 2006) - Michael Jahn: Fixed FTP_SKIP_PASV_IP and FTP_USE_EPSV to "do right" when used on FTP thru HTTP proxy. Fixed PROXYTUNNEL to work fine when you do ftp through a proxy. It would previously overwrite internal memory and cause unpredicted behaviour! Daniel (11 January 2006) - I decided to document the "secret option" here now, as I've received *NO* feedback at all on my mailing list requests from November 2005: I'm looking for feedback and comments. I added some experimental code the other day, that allows a libcurl user to select what method libcurl should use to reach a file on a FTP(S) server. This functionality is available in CVS code and in recent daily snapshots. Let me explain... The current name for the option is CURLOPT_FTP_FILEMETHOD (--ftp-method for the command line tool) and you set it to a long (there are currenly no defines for the argument values, just plain numericals). You can set three different "methods" that do this: 1 multicwd - like today, curl will do a single CWD operation for each path part in the given URL. For deep hierarchies this means very many commands. This is how RFC1738 says it should be done. This is the default. 2 nocwd - no CWD at all is done, curl will do SIZE, RETR, STOR etc and give a full path to the server. 3 singlecwd - make one CWD with the full target directory and then operate on the file "normally". (With the command line tool you do --ftp-method [METHOD], where [METHOD] is one of "multicwd", "nocwd" or "singlecwd".) What feedback I'm interested in: 1 - Do they work at all? Do you find servers where one of these don't work? 2 - What would proper names for the option and its arguments be, if we consider this feature good enough to get included and documented in upcoming releases? 3 - Should we make libcurl able to "walk through" these options in case of (path related) failures, or should it fail and let the user redo any possible retries? (This option is not documented in any man page just yet since I'm not sure these names will be used or if the functionality will end up exactly like this. And for the same reasons we have no test cases for these yet.) Daniel (10 January 2006) - When using a bad path over FTP, as in when libcurl couldn't CWD into all given subdirs, libcurl would still "remember" the full path as if it is the current directory libcurl is in so that the next curl_easy_perform() would get really confused if it tried the same path again - as it would not issue any CWD commands at all, assuming it is already in the "proper" dir. Starting now, a failed CWD command sets a flag that prevents the path to be "remembered" after returning. Daniel (7 January 2006) - Michael Jahn fixed so that the second CONNECT when doing FTP over a HTTP proxy actually used a new connection and not sent the second request on the first socket! Daniel (6 January 2006) - Alexander Lazic made the buildconf run the buildconf in the ares dir if that is present instead of trying to mimic that script in curl's buildconf script. Daniel (3 January 2006) - Andres Garcia made the TFTP test server build with mingw. Daniel (16 December 2005) - Jean Jacques Drouin pointed out that you could only have a user name or password of 127 bytes or less embedded in a URL, where actually the code uses a 255 byte buffer for it! Modified now to use the full buffer size. Daniel (12 December 2005) - Dov Murik corrected the HTTP_ONLY define to disable the TFTP support properly Version 7.15.1 (7 December 2005) Daniel (6 December 2005) - Full text here: http://curl.haxx.se/docs/adv_20051207.html Pointed out by Stefan Esser. VULNERABILITY libcurl's URL parser function can overflow a malloced buffer in two ways, if given a too long URL. These overflows happen if you 1 - pass in a URL with no protocol (like "http://") prefix, using no slash and the string is 256 bytes or longer. This leads to a single zero byte overflow of the malloced buffer. 2 - pass in a URL with only a question mark as separator (no slash) between the host and the query part of the URL. This leads to a single zero byte overflow of the malloced buffer. Both overflows can be made with the same input string, leading to two single zero byte overwrites. The affected flaw cannot be triggered by a redirect, but the long URL must be passed in "directly" to libcurl. It makes this a "local" problem. Of course, lots of programs may still pass in user-provided URLs to libcurl without doing much syntax checking of their own, allowing a user to exploit this vulnerability. There is no known exploit at the time of this writing. Daniel (2 December 2005) - Jamie Newton pointed out that libcurl's file:// code would close() a zero file descriptor if given a non-existing file. Daniel (24 November 2005) - Doug Kaufman provided a set of patches to make curl build fine on DJGPP again using configure. - Yang Tse provided a whole series of patches to clear up compiler warnings on MSVC 6. Daniel (17 November 2005) - I extended a patch from David Shaw to make libcurl _always_ provide an error string in the given error buffer to address the flaw mention on 21 sep 2005. Daniel (16 November 2005) - Applied Albert Chin's patch that makes the libcurl.pc pkgconfig file get installed on 'make install' time. Daniel (14 November 2005) - Quagmire reported that he needed to raise a NTLM buffer for SSPI to work properly for a case, and so we did. We raised it even for non-SSPI builds but it should not do any harm. http://curl.haxx.se/bug/view.cgi?id=1356715 - Jan Kunder's debian bug report http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=338680 identified a weird error message for when you try to upload a file and the requested directory doesn't exist on the target server. - Yang Tse fixed compiler warnings in lib/ssluse.c with OpenSSL 0.9.8 and in lib/memdebug.h that showed up in his msvc builds. Daniel (13 November 2005) - Debian bug report 338681 by Jan Kunder: make curl better detect and report bad limit-rate units: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=338681 Now curl will return error if a bad unit is used. - Thanks to this nice summary of poll() implementations: http://www.greenend.org.uk/rjk/2001/06/poll.html and further tests by Eugene Kotlyarov, we now know that cygwin's poll returns only POLLHUP on remote connectin closure so we check for that case (too) and re-enable poll for cygwin builds. Daniel (12 November 2005) - Eugene Kotlyarov found out that cygwin's poll() function isn't doing things right: http://curl.haxx.se/mail/archive-2005-11/0045.html so we now disable poll() and use select() on cygwin too (we already do the same choice on Mac OS X) - Dima Barsky patched problem #1348930: the GnuTLS code completely ignored client certificates! (http://curl.haxx.se/bug/view.cgi?id=1348930). Daniel (10 November 2005) - David Lang fixed IPv6 support for TFTP! - Introducing range stepping to the curl globbing support. Now you can specify step counter by adding :[num] within the brackets when specifying a range: [1-100:10] [a-z:2] If no step counter is set, it defaults to 1 as before: [1-100] [d-h] Daniel (8 November 2005) - Removed the use of AI_CANONNAME in the IPv6-enabled resolver functions since we really have no use for reverse lookups of the address. I truly hope these are the last reverse lookups we had lingering in the code! - Dmitry Bartsevich discovered some issues in compatibilty of SSPI-enabled version of libcurl with different Windows versions. Current version of libcurl imports SSPI functions from secur32.dll. However, under Windows NT 4.0 these functions are located in security.dll, under Windows 9x - in secur32.dll and Windows 2000 and XP contains both these DLLs (security.dll just forwards calls to secur32.dll). Dmitry's patch loads proper library dynamically depending on Windows version. Function InitSecurityInterface() is used to obtain pointers to all of SSPI function in one structure. Daniel (31 October 2005) - Vilmos Nebehaj improved libcurl's LDAP abilities: The LDAP code in libcurl can't handle LDAP servers of LDAPv3 nor binary attributes in LDAP objects. So, I made a quick patch to address these problems. The solution is simple: if we connect to an LDAP server, first try LDAPv3 (which is the preferred protocol as of now) and then fall back to LDAPv2. In case of binary attributes, we first convert them to base64, just like the openldap client does. It uses ldap_get_values_len() instead of ldap_get_values() to be able to retrieve binary attributes correctly. I defined the necessary LDAP macros in lib/ldap.c to be able to compile libcurl without the presence of libldap Daniel (27 October 2005) - Nis Jorgensen filed bug report #1338648 (http://curl.haxx.se/bug/view.cgi?id=1338648) which really is more of a feature request, but anyway. It pointed out that --max-redirs did not allow it to be set to 0, which then would return an error code on the first Location: found. Based on Nis' patch, now libcurl supports CURLOPT_MAXREDIRS set to 0, or -1 for infinity. Added test case 274 to verify. - tommink[at]post.pl reported in bug report #1337723 (http://curl.haxx.se/bug/view.cgi?id=1337723) that curl could not upload binary data from stdin on Windows if the data contained control-Z (hex 1a) since that is treated as end-of-file when read in text mode. Gisle Vanem pointed out the fix, and I made both -T and --data-binary take advantage of it. - Jaz Fresh pointed out that if you used "-r [number]" as was wrongly described in the man page, curl would send an invalid HTTP Range: header. The correct way would be to use "-r [number]-" or even "-r -[number]". Starting now, curl will warn if this is discovered, and automatically append a dash to the range before passing it to libcurl. Daniel (25 October 2005) - Amol Pattekar reported a bug with great detail and a fine example in bug #1326306 (http://curl.haxx.se/bug/view.cgi?id=1326306). When using the multi interface and connecting to a host with multiple IP addresses, and one of the addresses fails to connect (the server must exist and respond, just not accept connections) libcurl leaks a socket descriptor. Thanks to the fine report, I could find and fix this. Daniel (22 October 2005) - Dima Barsky reported a problem with GnuTLS-enabled libcurl in bug report #1334338 (http://curl.haxx.se/bug/view.cgi?id=1334338). When reading an SSL stream from a server and the server requests a "rehandshake", the current code simply returns this as an error. I have no good way to test this, but I've added a crude attempt of dealing with this situation slightly better - it makes a blocking handshake if this happens. Done like this because fixing this the "proper" way (that would handshake asynchronously) will require quite some work and I really need a good way to test this to do such a change. Daniel (21 October 2005) - "Ofer" reported a problem when libcurl re-used a connection and failed to do it, it could then accidentally actually crash. Presumably, this concerns FTP connections. http://curl.haxx.se/bug/view.cgi?id=1330310 - Temprimus improved the MSVC makefile so that the static debug SSL libs are linked to the executable and not to the libcurld.lib http://curl.haxx.se/bug/view.cgi?id=1326676 - Bradford Bruce made the windows resolver code properly return CURLE_COULDNT_RESOLVE_PROXY and CURLE_COULDNT_RESOLVE_HOST on resolving errors (as documented). Daniel (20 October 2005) - Dave Dribin made libcurl understand and handle cases when the server (wrongly) sends *two* WWW-Authenticate headers for Digest. While this should never happen in a sane world, libcurl previously got into an infinite loop when this occurred. Dave added test 273 to verify this. - Temprimus improved the MSVC makefile: "makes a build option available so if you set rtlibcfg=static for the make, then it would build with /MT. The default behaviour is /MD (the original)." http://curl.haxx.se/bug/view.cgi?id=1326665 Daniel (14 October 2005) - Reverted the LIBCURL_VERSION_NUM change from October 6. As Dave Dribin reported, the define is used by the configure script and is assumed to use the 0xYYXXZZ format. This made "curl-config --vernum" fail in the 7.15.0 release version. Version 7.15.0 (13 October 2005) Daniel (12 October 2005) - Michael Sutton of iDEFENSE reported and I fixed a securitfy flaw in the NTLM code that would overflow a buffer if given a too long user name or domain name. This would happen if you enable NTLM authentication and either A - pass in a user name and domain name to libcurl that together are longer than 192 bytes B - allow (lib)curl to follow HTTP "redirects" (Location: and the appropriate HTTP 30x response code) and the new URL contains a URL with a user name and domain name that together are longer than 192 bytes See http://curl.haxx.se/docs/security.html for further details and updates Daniel (5 October 2005) - Darryl House reported a problem with using -z to download files from FTP. It turned out that if the given time stamp was exact the same as the remote time stamp, the file would still wrongly be downloaded. Added test case 272 to verify. Daniel (4 October 2005) - Domenico Andreoli fixed a man page malformat and removed odd (0xa0) bytes from the configure script. - Michael Wallner reported that the date parser had wrong offset stored for the MEST and CEST time zones. Daniel (27 September 2005) - David Yan filed bug #1299181 (http://curl.haxx.se/bug/view.cgi?id=1299181) that identified a silly problem with Content-Range: headers with the 'bytes' keyword written in a different case than all lowercase! It would cause a segfault! - TJ Saunders of the proftpd project identified and pointed out problems with the modified FTPS negotiation change of August 19 2005. Thus, we revert the change back to pre-7.14.1 status. Daniel (21 September 2005) - Fixed "cut off" sentence in the libcurl-tutorial man page: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=329305 - Clarified in the curl_easy_setopt man page what the default CURLOPT_WRITEFUNCTION and CURLOPT_WRITEDATA mean: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=329311 - Clarified in the curl_easy_setopt man page that CURLOPT_ERRORBUFFER sometimes doesn't fill in the buffer even though it is supposed to: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=329313 - When CURLE_URL_MALFORMAT is returned due to a missing URL, it now has an error string set. Daniel (19 September 2005) - Dmitry Bartsevich made the SSPI support work on Windows 9x as well. Daniel (15 September 2005) - Added a TFTP server to the test suite and made the test suite capable of using it. Daniel (7 September 2005) - Ben Madsen's detailed reports that funnily enough only occurred with certain glibc versions turned out to be curl using an already closed file handle during certain conditions (like when saving FTP server "headers"). - Scott Davis helped me track down a problem in the test HTTP server that made test case 56 wrongly fail at times. It turned out it was due to the server finding the end of a chunked-encoded POST too early. Daniel (6 September 2005) - Now curl warns if an unknown variable is used in the -w/--writeout argument. Daniel (4 September 2005) - I applied Nicolas François' man page patch he posted to the Debian bug tracker. It corrected two lines that started with apostrophes, which isn't legal nroff format. http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=326511 - Added --ftp-skip-pasv-ip to the command line tool, that sets the new CURLOPT_FTP_SKIP_PASV_IP option. It makes libcurl re-use the control connection's IP address when setting up the data connection instead of extractting the IP address from the PASV response. It has turned out this feature is frequently needed by people to circumvent silly servers and silly firewalls, especially when FTPS is used and the PASV command-response is sent encrtyped. Sponsored by CU*Answers Daniel (1 September 2005) - John Kelly added TFTP support to libcurl. A bunch of new error codes was added. TODO: add them to docs. add TFTP server to test suite. add TFTP to list of protocols whereever those are mentioned. Version 7.14.1 (1 September 2005) Daniel (29 August 2005) - Kevin Lussier pointed out a problem with curllib.dsp and how to fix it. - Igor Polyakov fixed a rather nasty problem with the threaded name resolver for Windows, that could lead to an Access Violation when the multi interface was used due to an issue with how the resolver thread was and was not terminated. - Simon Josefsson brought a patch that allows curl to get built to use GNU GSS instead of MIT/Heimdal for GSS capabilities. Daniel (24 August 2005) - Toby Peterson added CURLOPT_IGNORE_CONTENT_LENGTH to the library, accessible from the command line tool with --ignore-content-length. This will make it easier to download files from Apache 1.x (and similar) servers that are still having problems serving files larger than 2 or 4 GB. When this option is enabled, curl will simply have to wait for the server to close the connection to signal end of transfer. I wrote test case 269 that runs a simple test to verify that this works. - (Trying hard to exclude emotions now.) valgrind version 3 suddenly renamed the --logfile command line option to --log-file, and thus the test script valgrind autodetection now has yet another version check to do and then it alters the valgrind command line accordingly. - Fixed CA cert verification using GnuTLS with the default bundle, which previously failed due to GnuTLS not allowing x509 v1 CA certs by default. Ralph Mitchell reported. Daniel (19 August 2005) - Norbert Novotny had problems with FTPS and he helped me work out a patch that made curl run fine in his end. The key was to make sure we do the SSL/TLS negotiation immediately after the TCP connect is done and not after a few other commands have been sent like we did previously. I don't consider this change necessary to obey the standards, I think this server is pickier than what the specs allow it to be, but I can't see how this modified libcurl code can add any problems to those who are interpreting the standards more liberally. Daniel (17 August 2005) - Jeff Pohlmeyer found out that if you ask libcurl to load a cookiefile (with CURLOPT_COOKIEFILE), add a cookie (with CURLOPT_COOKIELIST), tell it to write the result to a given cookie jar and then never actually call curl_easy_perform() - the given file(s) to read was never read but the output file was written and thus it caused a "funny" result. - While doing some tests for the bug above, I noticed that Firefox generates large numbers (for the expire time) in the cookies.txt file and libcurl didn't treat them properly. Now it does. Daniel (15 August 2005) - Added more verbose "warning" messages to the curl client for cases where it fails to open/read files etc to help users diagnose why it doesn't do what you'd expect it to. Converted lots of old messages to use the new generic function I wrote for this purpose. Daniel (13 August 2005) - James Bursa identified a libcurl HTTP bug and a good way to repeat it. If a site responds with bad HTTP response that doesn't contain any header at all, only a response body, and the write callback returns 0 to abort the transfer, it didn't have any real effect but the write callback would be called once more anyway. Daniel (12 August 2005) - Based on Richard Clayton's reports, I found out that using curl -d @filename when 'filename' was not possible to access made curl use a GET request instead. - The time condition illegal syntax warning is now inhibited if -s is used. Daniel (10 August 2005) - Mario Schroeder found out that one of the debug callbacks calls that regards SSL data with the CURLINFO_TEXT type claimed that the data was one byte larger than it actually is, thus falsely telling the application that the terminating zero was part of the data. Daniel (9 August 2005) - Christopher R. Palmer fixed the offsets used for date parsings when the time zone name of a daylight savings time was used. For example, PDT vs PDS. This flaw was introduced with the new date parser (11 sep 2004 - 7.12.2). Fortunately, no web server or cookie string etc should be using such time zone names thus limiting the effect of this bug. Daniel (8 August 2005) - Jon Grubbs filed bug report #1249962 (http://curl.haxx.se/bug/view.cgi?id=1249962) which identified a problem with NTLM on a HTTP proxy if an FTP URL was given. libcurl now properly switches to pure HTTP internally when an HTTP proxy is used, even for FTP URLs. The problem would also occur with other multi-pass auth methods. Daniel (7 August 2005) - When curl is built with GnuTLS, curl-config didn't include "SSL" when --features was used. Daniel (28 July 2005) - If any of the options CURLOPT_HTTPGET, CURLOPT_POST and CURLOPT_HTTPPOST is set to 1, CURLOPT_NOBODY will now automatically be set to 0. Daniel (27 July 2005) - Dan Fandrich changes over the last week: fixed numerous minor configure option parsing flaws: --without-gnutls, --without-spnego --without-gssapi and --without-krb4. Spellfixed several error messages. - Peteris Krumins added CURLOPT_COOKIELIST and CURLINFO_COOKIELIST, which is a simple interface to extracting and setting cookies in libcurl's internal "cookie jar". See the new cookie_interface.c example code. Daniel (13 July 2005) - Diego Casorran provided patches to make curl build fine on Amiga again. Daniel (12 July 2005) - Adrian Schuur added trailer support in the chunked encoding stream. The trailer is then sent to the normal header callback/stream. I wrote up test case 266 to verify the basic functionality. Do note that test case 34 contains a flawed chunked encoding stream that still works the same. Daniel (5 July 2005) - Gisle Vanem came up with a nice little work-around for bug #1230118 (http://curl.haxx.se/bug/view.cgi?id=1230118). It seems the Windows (MSVC) libc time functions may return data one hour off if TZ is not set and automatic DST adjustment is enabled. This made curl_getdate() return wrong value, and it also concerned internal cookie expirations etc. Daniel (4 July 2005) - Andrew Bushnell provided enough info for me to tell that we badly needed to fix the CONNECT authentication code with multi-pass auth methods (such as NTLM) as it didn't previously properly ignore response-bodies - in fact it stopped reading after all response headers had been received. This could lead to libcurl sending the next request and reading the body from the first request as response to the second request. (I also renamed the function, which wasn't strictly necessary but...) The best fix would to once and for all make the CONNECT code use the ordinary request sending/receiving code, treating it as any ordinary request instead of the special-purpose function we have now. It should make it better for multi-interface too. And possibly lead to less code... Added test case 265 for this. It doesn't work as a _really_ good test case since the test proxy is too stupid, but the test case helps when running the debugger to verify. Daniel (30 June 2005) - Dan Fandrich improved the configure script's ability to figure out what kind of strerror_r() API that is used when cross-compiling. If __GLIB__ is defined, it assumes the glibc API. If not, it issues a notice as before that the user needs to manually edit lib/config.h for this. Daniel (23 June 2005) - David Shaw's fix that unifies proxy string treatment so that a proxy given with CURLOPT_PROXY can use a http:// prefix and user + password. The user and password fields are now also URL decoded properly. Test case 264 added to verify. Daniel (22 June 2005) - David Shaw updated libcurl.m4 Daniel (14 June 2005) - Gisle Vanem fixed a potential thread handle leak. Bug report #1216500 (http://curl.haxx.se/bug/view.cgi?id=1216500). Comment in http://curl.haxx.se/mail/lib-2005-06/0059.html Daniel (13 June 2005) - Made buildconf run libtoolize in the ares dir too (inspired by Tupone's reverted patch). Daniel (9 June 2005) - Incorporated Tupone's findtool fix in buildconf (slightly edited) - Incorporated Tupone's head -n fix in buildconf. Daniel (8 June 2005) - Reverted Tupone's patch again, it broke numerous autobuilds. Let's apply it in pieces, one by one and see what we need to adjust to work all over. Daniel (6 June 2005) - Tupone Alfredo fixed three problems in buildconf: 1) findtool does look per tool in PATH and think ./perl is the perl executable, while is just a local directory (I have . in the PATH) 2) I got several warning for head -1 deprecated in favour of head -n 1 3) ares directory is missing some file (missing is missing :-) ) because automake and friends is not run. Daniel (3 June 2005) - Added docs/libcurl/getinfo-times, based on feedback from 'Edi': http://curl.haxx.se/feedback/display.cgi?id=11178325798299&support=yes - Andres Garcia provided yet another text mode patch for several test cases so that they do text comparisions better on Windows (newline-wise). Daniel (1 June 2005) - The configure check for c-ares now adds the cares lib before the other libs, to make it build fine with mingw. Inspired by Tupone Alfredo's bug report and patch: http://curl.haxx.se/bug/view.cgi?id=1212940 Daniel (31 May 2005) - Todd Kulesza reported a flaw in the proxy option, since a numerical IPv6 address was not possible to use. It is now, but requires it written RFC2732-style, within brackets - which incidently is how you enter numerical IPv6 addresses in URLs. Test case 263 added to verify. Daniel (30 May 2005) - Eric Cooper reported about a problem with HTTP servers that responds with binary zeroes within the headers. They confused libcurl to do wrong so the downloaded headers become incomplete. The fix is now verified with test case 262. http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=310948 Daniel (25 May 2005) - Fixed problems with the test suite, and in particular the FTP test cases since it previously was failing every now and then in a nonsense manner. - --trace-time now outputs the full microsecond, all 6 digits. Daniel (24 May 2005) - Andres Garcia provided a text mode patch for several test cases so that they do text comparisions better on Windows (newline-wise). - Any 2xx response (and not just 200) is now considered a fine response to TYPE, as some servers obviously sends a 226 there. Added test case 261 to verify. Based on a question/report by Georg Wicherski. Daniel (20 May 2005) - Improved runtests.pl to allow stdout tests to be mode=text as well, just as file comparisons already supports. Added this info to the FILEFORMAT docs. Daniel (18 May 2005) - John McGowan identified a problem in bug report #1204435 (http://curl.haxx.se/bug/view.cgi?id=1204435) with malformed URLs like "http://somehost?data" as it added a slash too much in the request ("GET /?data/"...). Added test case 260 to verify. - The configure check for strerror_r() failed to detect the proper API at times, like on HP-UX 10.20. Then lib/strerror.c badly assumed the glibc version if the posix define wasn't set (since it _had_ found a strerror_r). Daniel (16 May 2005) - The gmtime_r() function in HP-UX 10.20 is broken. About 13 test cases fail due to this. There's now a configure check that attempts to detect the bad function and not use it on such systems. Version 7.14.0 (16 May 2005) Daniel (13 May 2005) - Grigory Entin reported that curl's configure detects a fine poll() for Mac OS X 10.4 (while 10.3 or later detected a "bad" one), but the executable doesn't work as good as if built without poll(). I've adjusted the configure to always skip the fine-poll() test on Mac OS X (darwin). Daniel (12 May 2005) - When doing a second request (after a disconnect) using the same easy handle, over a proxy that uses NTLM authentication, libcurl failed to use NTLM again properly (the auth method was accidentally reset to the same as had been set for host auth, which defaults to Basic). Bug report #1200661 (http://curl.haxx.se/bug/view.cgi?id=1200661) identified the the problem and the fix. - If -z/--time-cond is used with an invalid date syntax, this is no longer silently discarded. Instead a proper warning message is diplayed that informs about it. But it still continues without the condition. Version 7.14.0-pre2 (11 May 2005) Daniel (11 May 2005) - Starting now, libcurl sends a little different set of headers in its default HTTP requests: A) Normal non-proxy HTTP: - no more "Pragma: no-cache" (this only makes sense to proxies) B) Non-CONNECT HTTP request over proxy: - "Pragma: no-cache" is used (like before) - "Proxy-Connection: Keep-alive" (for older style 1.0-proxies) C) CONNECT HTTP request over proxy: - "Host: [name]:[port]" - "Proxy-Connection: Keep-alive" The A) case is mostly to reduce the default header size and remove a pointless header. The B) is to address (rare) problems with HTTP 1.0 proxies The C) headers are both to address (rare) problems with some proxies. The code in libcurl that deals with CONNECT requests need a rewrite, but it feels like a too big a job for me to do now. Details are added in the code comments for now. Updated a large amount of test cases to reflect the news. Daniel (10 May 2005) - Half-baked attempt to bail out if select() returns _only_ errorfds when the transfer is in progress. An attempt to fix Allan's problem. See http://curl.haxx.se/mail/lib-2005-05/0073.html and the rest of that thread for details. I'm still not sure this is the right fix, but... Version 7.14.0-pre1 (9 May 2005) Daniel (2 May 2005) - Sort of "fixed" KNOWN_BUGS #4: curl now builds IPv6 enabled on AIX 4.3. At least it should no longer cause a compiler error. However, it does not have AI_NUMERICHOST so we cannot getaddrinfo() any numerical addresses with it (we use that for FTP PORT/EPRT)! So, I modified the configure check that checks if the getaddrinfo() is working, to use AI_NUMERICHOST since then it'll fail on AIX 4.3 and it will automatically build with IPv6 support disabled. - Added --trace-time that when used adds a time stamp to each trace line that --trace, --trace-ascii and --verbose output. I also made the '>' display separate each line on the linefeed so that HTTP requests etc look nicer in the -v output. - Made curl recognize the environment variables Lynx (and others?) support for pointing out the CA cert path/file: SSL_CERT_DIR and SSL_CERT_FILE. If CURL_CA_BUNDLE is not set, they are checked afterwards. Like before: on windows if none of these are set, it checks for the ca cert file like this: 1. application's directory 2. current working directory 3. Windows System directory (e.g. C:\windows\system32) 4. Windows Directory (e.g. C:\windows) 5. all directories along %PATH% Daniel (1 May 2005) - The runtests.pl script now starts test servers by doing fork() and exec() instead of the previous approach. This is less complicated and should hopefully lead to less "leaked" servers (servers that aren't stopped properly when the tests are stopped). - Alexander Zhuravlev found a case when you did "curl -I [URL]" and it complained on the chunked encoding, even though a HEAD should never return a body and thus it cannot be a chunked-encoding problem! Daniel (30 April 2005) - Alexander Zhuravlev found out that (lib)curl SIGSEGVed when using --interface on an address that can't be bound. Daniel (28 April 2005) - Working on fixing up test cases to mark sections as 'mode=text' for things that curl writes as text files, since then they can get different line endings depending on OS. Andrés García helps me work this out. Did lots of other minor tweaks on the test scripts to work better and more reliably find test servers and also kill test servers. - Dan Fandrich pointed out how the runtests.pl script killed the HTTP server instead of the HTTPS server when closing it down. Daniel (27 April 2005) - Paul Moore made curl check for the .curlrc file (_curlrc on windows) on two more places. First, CURL_HOME is a new environment variable that is used instead of HOME if it is set, to point out where the default config file lives. If there's no config file in the dir pointed out by one of the environment variables, the Windows version will instead check the same directory the executable curl is located in. Daniel (26 April 2005) - Cory Nelson's work on nuking compiler warnings when building on x64 with VS2005. Daniel (25 April 2005) - Fred New reported a bug where we used Basic auth and user name and password in .netrc, and when following a Location: the subsequent requests didn't properly use the auth as found in the netrc file. Added test case 257 to verify my fix. - Based on feedback from Cory Nelson, I added some preprocessor magic in */setup.h and */config-win32.h to build fine with VS2005 on x64. Daniel (23 April 2005) - Alex Suykov made the curl tool now assume that uploads using HTTP:// or HTTPS:// are the only ones that show output and thus motivates a switched off progress meter if the output is sent to the terminal. This makes FTP uploads without '>', -o or -O show the progress meter. Daniel (22 April 2005) - Dave Dribin's MSVC makefile fix: set CURL_STATICLIB when it builds static library variants. - Andres Garcia fixed configure to set the proper define when building static libcurl on windows. - --retry-delay didn't work. Daniel (18 April 2005) - Olivier reported that even though he used CURLOPT_PORT, libcurl clearly still used the default port. He was right. I fixed the problem and added the test cases 521, 522 and 523 to verify the fix. - Toshiyuki Maezawa reported that when doing a POST with a read callback, libcurl didn't properly send an Expect: 100-continue header. It does now. - I committed by mig change in the test suite's FTP server that moves out all socket/TCP code to a separate C program named sockfilt. And added 4 new test cases for FTP over IPv6. Daniel (8 April 2005) - Cory Nelson reported a problem with a HTTP server that responded with a 304 response containing an "illegal" Content-Length: header, which was not properly ignored by libcurl. Now it is. Test case 249 verifies. Daniel (7 April 2005) - Added ability to build and run with GnuTLS as an alternative to OpenSSL for the secure layer. configure --with-gnutls enables with. Note that the previous OpenSSL check still has preference and if it first detects OpenSSL, it will not check for GnuTLS. You may need to explictly diable OpenSSL with --without-ssl. This work has been sponsored by The Written Word. Daniel (5 April 2005) - Christophe Legry fixed the post-upload check for FTP to not complain if the upload was skipped due to a time-condition as set with CURLOPT_TIMECONDITION. I added test case 247 and 248 to verify. Version 7.13.2 (5 April 2005) Daniel (4 April 2005) - Marcelo Juchem fixed the MSVC makefile for libcurl - Gisle Vanem fixed a crash in libcurl, that could happen if the easy handle was killed before the threading resolver (windows only) still hadn't completed. - Hardeep Singh reported a problem doing HTTP POST with Digest. (It was actually also affecting NTLM and Negotiate.) It turned out that if the server responded with 100 Continue before the initial 401 response, libcurl didn't take care of the response properly. Test case 245 and 246 added to verify this. Daniel (30 March 2005) - Andres Garcia modified the configure script to check for libgdi32 before libcrypto, to make the SSL check work fine on msys/mingw. Daniel (29 March 2005) - Tom Moers identified a flaw when you sent a POST with Digest authentication, as in the first request when curl sends a POST with Content-Length: 0, it still forcibly closed the connection before doing the next step in the auth negotiation. - Jesper Jensen found out that FTP-SSL didn't work since my FTP rewrite. Fixing that was easy, but it also revealed a much worse problem: the FTP server response reader function didn't properly deal with reading responses in multiple tiny chunks properly! I modified the FTP server to allow it to produce such split-up responses to make sure curl deals with them as it should. - Based on Augustus Saunders' comments and findings, the HTTP output auth function was fixed to use the proper proxy authentication when multiple ones are accepted. test 239 and test 243 were added to repeat the problems and verify the fixes. --proxy-anyauth was added to the curl tool Daniel (16 March 2005) - Tru64 and some IRIX boxes seem to not like test 237 as it is. Their inet_addr() functions seems to use &255 on all numericals in a ipv4 dotted address which makes a different failure... Now I've modified the ipv4 resolve code to use inet_pton() instead in an attempt to make these systems better detect this as a bad IP address rather than creating a toally bogus address that is then passed on and used. Daniel (15 March 2005) - Dan Fandrich made the code properly use the uClibc's version of inet_ntoa_r() when built with it. - Added test 237 and 238: test EPSV and PASV response handling when they get well- formated data back but using illegal values. In 237 PASV gets an IP address that is way bad. In 238 EPSV gets a port that is way out of range. Daniel (14 March 2005) - Added a few missing features to the curl-config --features list - Modified testcurl.pl to now offer 1 - command line options for all info it previously only read from file: --name, --email, --desc and --configure 2 - --nocvsup makes it not attempt to do cvs update 3 - --crosscompile informs it and makes it not attempt things it can't do - Fixed numerous win32 compiler warnings. - Removed the lib/security.h file since it shadowed the mingw/win32 header with the same name which is needed for SSPI builds. The contents of the former security.h is now i krb4.h - configure --enable-sspi now enables SSPI in the build. It only works for windows builds (including cross-compiles for windows). Daniel (12 March 2005) - David Houlder added --form-string that adds that string to a multipart formpost part, without special characters having special meanings etc like --form features. Daniel (11 March 2005) - curl_version_info() returns the feature bit CURL_VERSION_SSPI if it was built with SSPI support. - Christopher R. Palmer made it possible to build libcurl with the USE_WINDOWS_SSPI on Windows, and then libcurl will be built to use the native way to do NTLM. SSPI also allows libcurl to pass on the current user and its password in the request. Daniel (9 March 2005) - Dan F improved the SSL lib setup in configure. - Nodak Sodak reported a crash when using a SOCKS4 proxy. - Jean-Marc Ranger pointed out an embarassing debug printf() leftover in the multi interface code. - Adjusted the man page for the curl_getdate() return value for dates after year 2038. For 32 bit time_t it returns 0x7fffffff but for 64bit time_t it returns either the correct value or even -1 on some systems that still seem to not deal with this properly. Tor Arntsen found a 64bit AIX system for us that did the latter. Gwenole Beauchesne's Mandrake patch put the lights on this problem in the first place. Daniel (8 March 2005) - Dominick Meglio reported that using CURLOPT_FILETIME when transferring a FTP file got a Last-Modified: header written to the data stream, corrupting the actual data. This was because some conditions from the previous FTP code was not properly brought into the new FTP code. I fixed and I added test case 520 to verify. (This bug was introduced in 7.13.1) - Dan Fandrich fixed the configure --with-zlib option to always consider the given path before any standard paths. Daniel (6 March 2005) - Randy McMurchy was the first to report that valgrind.pm was missing from the release archive and thus 'make test' fails. Daniel (5 March 2005) - Dan Fandrich added HAVE_FTRUNCATE to several config-*.h files. - Added test case 235 that makes a resumed upload of a file that isn't present on the remote side. This then converts the operation to an ordinary STOR upload. This was requested/pointed out by Ignacio Vazquez-Abrams. It also proved (and I fixed) a bug in the newly rewritten ftp code (and present in the 7.13.1 release) when trying to resume an upload and the servers returns an error to the SIZE command. libcurl then loops and sends SIZE commands infinitely. - Dan Fandrich fixed a SSL problem introduced on February 9th that made libcurl attempt to load the whole random file to seed the PRNG. This is really bad since this turns out to be using /dev/urandom at times... Version 7.13.1 (4 March 2005) Daniel (4 March 2005) - Dave Dribin made it possible to set CURLOPT_COOKIEFILE to "" to activate the cookie "engine" without having to provide an empty or non-existing file. - Rene Rebe fixed a -# crash when more data than expected was retrieved. Daniel (22 February 2005) - NTLM and ftp-krb4 buffer overflow fixed, as reported here: http://www.securityfocus.com/archive/1/391042 and the CAN report here: http://cve.mitre.org/cgi-bin/cvename.cgi?name=CAN-2005-0490 If these security guys were serious, we'd been notified in advance and we could've saved a few of you a little surprise, but now we weren't. Daniel (19 February 2005) - Ralph Mitchell reported a flaw when you used a proxy with auth, and you requested data from a host and then followed a redirect to another host. libcurl then didn't use the proxy-auth properly in the second request, due to the host-only check for original host name wrongly being extended to the proxy auth as well. Added test case 233 to verify the flaw and that the fix removed the problem. Daniel (18 February 2005) - Mike Dobbs reported a mingw build failure due to the lack of BUILDING_LIBCURL being defined when libcurl is built. Now this is defined by configure when mingw is used. Daniel (17 February 2005) - David in bug report #1124588 found and fixed a socket leak when libcurl didn't close the socket properly when returning error due to failing localbind Daniel (16 February 2005) - Christopher R. Palmer reported a problem with HTTP-POSTing using "anyauth" that picks NTLM. Thanks to David Byron letting me test NTLM against his servers, I could quickly repeat and fix the problem. It turned out to be: When libcurl POSTs without knowing/using an authentication and it gets back a list of types from which it picks NTLM, it needs to either continue sending its data if it keeps the connection alive, or not send the data but close the connection. Then do the first step in the NTLM auth. libcurl didn't send the data nor close the connection but simply read the response-body and then sent the first negotiation step. Which then failed miserably of course. The fixed version forces a connection if there is more than 2000 bytes left to send. Daniel (14 February 2005) - The configure script didn't check for ENGINE_load_builtin_engines() so it was never used. Daniel (11 February 2005) - Removed all uses of strftime() since it uses the localised version of the week day names and month names and servers don't like that. Daniel (10 February 2005) - Now the test script disables valgrind-testing when the test suite runs if libcurl is built shared. Otherwise valgrind only tests the shell that runs the wrapper-script named 'curl' that is a front-end to curl in this case. This should also fix the huge amount of reports of false positives when valgrind has identified leaks in (ba)sh and not in curl and people report that as curl bugs. Bug report #1116672 is one example. Also, the valgrind report parser has been adapted to check that at least one of the sources in a stack strace is one of (lib)curl's source files or otherwise it will not consider the problem to concern (lib)curl. - Marty Kuhrt streamlined the VMS build. Daniel (9 February 2005) - David Byron fixed his SSL problems, initially mentioned here: http://curl.haxx.se/mail/lib-2005-01/0240.html. It turned out we didn't use SSL_pending() as we should. - Converted lots of FTP code to a statemachine, so that the multi interface doesn't block while communicating commands-responses with an FTP server. I've added a comment like BLOCKING in the code on all spots I could find where we still have blocking operations. When we change curl_easy_perform() to use the multi interface, we'll also be able to simplify the code since there will only be one "internal interface". While doing this, I've now made CURLE_FTP_ACCESS_DENIED separate from the new CURLE_LOGIN_DENIED. The first one is now access denied to a function, like changing directory or retrieving a file, while the second means that we were denied login. The CVS tag 'before_ftp_statemachine' was set just before this went in, in case of future need. - Gisle made the DICT code send CRLF and not just LF as the spec says so. Daniel (8 February 2005) - Gisle fixed problems when libcurl runs out of memory, and worked on making sure the proper error code is returned for those occations. Daniel (7 February 2005) - Maruko pointed out a problem with inflate decompressing exactly 64K contents. Daniel (5 February 2005) - Eric Vergnaud found a use of an uninitialised variable in the ftp when doing PORT on ipv6-enabled hosts. - David Byron pointed out we could use BUFSIZE to read data (in lib/transfer.c) instead of using BUFSIZE -1. Version 7.13.0 (1 February 2005) Daniel (31 January 2005) - Added Lars Nilsson's htmltitle.cc example Daniel (30 January 2005) - Fixed a memory leak when using the multi interface and the DO operation failed (as in test case 205). - Fixed a valgrind warning for file:// operations. - Fixed a valgrind report in the url globbing code for the curl command line tool. - Bugfixed the parser that scans the valgrind report outputs (in runtests.pl). I noticed that it previously didn't detect and report the "Conditional jump or move depends on uninitialised value(s)" error. When I fixed this, I caught a few curl bugs with it. And then I had to spend time to make the test suite IGNORE these errors when OpenSSL is used since it produce massive amounts of valgrind warnings (but only of the "Conditional..." kind it seems). So, if a test that requires SSL is run, it ignores the "Conditional..." errors, and you'll get a "valgrind PARTIAL" output instead of "valgrind OK". Daniel (29 January 2005) - Using the multi interface, and doing a requsted a re-used connection that gets closed just after the request has been sent failed and did not re-issue a request on a fresh reconnect like the easy interface did. Now it does! - Define CURL_MULTIEASY when building libcurl (lib/easy.c to be exact), to use my new curl_easy_perform() that uses the multi interface to run the request. It is a great testbed for the multi interface and I believe we shall do it this way for real in the future when we have a successor to curl_multi_fdset(). I've used this approach to detect and fix several of the recent multi-interfaces issues. - Adjusted the KNOWN_BUGS #17 fix a bit more since the FTP code also did some bad assumptions. - multi interface: when a request is denied due to "Maximum redirects followed" libcurl leaked the last Location: URL. - Connect failures with the multi interface was often returned as "connect() timed out" even though the reason was different. Daniel (28 January 2005) - KNOWN_BUGS #17 fixed. A DNS cache entry may not remain locked between two curl_easy_perform() invokes. It was previously unlocked at disconnect, which could mean that it remained locked between multiple transfers. The DNS cache may not live as long as the connection cache does, as they are separate. To deal with the lack of DNS (host address) data availability in re-used connections, libcurl now keeps a copy of the IP adress as a string, to be able to show it even on subsequent requests on the same connection. The problem could be made to appear with this stunt: 1. create a multi handle 2. add an easy handle 3. fetch a URL that is persistent (leaves the connection alive) 4. remove the easy handle from the multi 5. kill the multi handle 6. create a multi handle 7. add the same easy handle to the new multi handle 8. fetch a URL from the same server as before (re-using the connection) - Stephen More pointed out that CURLOPT_FTPPORT and the -P option didn't work when built ipv6-enabled. I've now made a fix for it. Writing test cases for custom port hosts turned too tricky so unfortunately there's none. Daniel (25 January 2005) - Ian Ford asked about support for the FTP command ACCT, and I discovered it is present in RFC959... so now (lib)curl supports it as well. --ftp-account and CURLOPT_FTP_ACCOUNT set the account string. (The server may ask for an account string after PASS have been sent away. The client responds with "ACCT [account string]".) Added test case 228 and 229 to verify the functionality. Updated the test FTP server to support ACCT somewhat. - David Shaw contributed a fairly complete and detailed autoconf test you can use to detect libcurl and setup variables for the protocols the installed libcurl supports: docs/libcurl/libcurl.m4 Daniel (21 January 2005) - Major FTP third party transfer overhaul. These four options are now obsolete: CURLOPT_SOURCE_HOST, CURLOPT_SOURCE_PATH, CURLOPT_SOURCE_PORT (this option didn't work before) and CURLOPT_PASV_HOST. These two options are added: CURLOPT_SOURCE_URL and CURLOPT_SOURCE_QUOTE. The target-side didn't use the proper path with RETR, and thus this only worked correctly in the login path (i.e without doing any CWD). The source- side still uses a wrong path, but the fix for this will need to wait. Verify the flaw by using a source URL with included %XX-codes. Made CURLOPT_FTPPORT control weather the target operation should use PORT (or not). The other side thus uses passive (PASV) mode. Updated the ftp3rdparty.c example source to use the updated options. Added support for a second FTP server in the test suite. Named... ftp2. Added test cases 230, 231 and 232 as a few first basic tests of very simple 3rd party transfers. Changed the debug output to include 'target' and 'source' when a 3rd party is being made, to make it clearer what commands/responses came on what connection. Added three new command line options: --3p-url, --3p-user and --3p-quote. Documented the command line options and the curl_easy_setopt options related to third party transfers. (Temporarily) disabled the ability to re-use an existing connection for the source connection. This is because it needs to force a new in case the source and target is the same host, and the host name check is trickier now when the source is identified with a full URL instead of a plain host name like before. TODO (short-term) for 3rd party transfers: quote support. The options are there, we need to add test cases to verify their functionality. TODO (long-term) for 3rd party transfers: IPv6 support (EPRT and EPSV etc) and SSL/TSL support. Daniel (20 January 2005) - Philippe Hameau found out that -Q "+[command]" didn't work, although some code was written for it. I fixed and added test case 227 to verify it. The curl.1 man page didn't mention the '+' so I added it. Daniel (19 January 2005) - Stephan Bergmann made libcurl return CURLE_URL_MALFORMAT if an FTP URL contains %0a or %0d in the user, password or CWD parts. (A future fix would include doing it for %00 as well - see KNOWN_BUGS for details.) Test case 225 and 226 were added to verify this - Stephan Bergmann pointed out two flaws in libcurl built with HTTP disabled: 1) the proxy environment variables are still read and used to set HTTP proxy 2) you couldn't disable http proxy with CURLOPT_PROXY (since the option was disabled). This is important since apps may want to disable HTTP proxy without actually knowing if libcurl was built to disable HTTP or not. Based on Stephan's patch, both these issues should now be fixed. Daniel (18 January 2005) - Cody Jones' enhanced version of Samuel Díaz García's MSVC makefile patch was applied. Daniel (16 January 2005) - Alex aka WindEagle pointed out that when doing "curl -v dictionary.com", curl assumed this used the DICT protocol. While guessing protocols will remain fuzzy, I've now made sure that the host names must start with "[protocol]." for them to be a valid guessable name. I also removed "https" as a prefix that indicates HTTPS, since we hardly ever see any host names using that. Daniel (13 January 2005) - Inspired by Martijn Koster's patch and example source at http://www.greenhills.co.uk/mak/gentoo/curl-eintr-bug.c, I now made the select() and poll() calls properly loop if they return -1 and errno is EINTR. glibc docs for this is found here: http://www.gnu.org/software/libc/manual/html_node/Interrupted-Primitives.html This last link says BSD doesn't have this "effect". Will there be a problem if we do this unconditionally? Daniel (11 January 2005) - Dan Torop cleaned up a few no longer used variables from David Phillips' select() overhaul fix. - Cyrill Osterwalder posted a detailed analysis about a bug that occurs when using a custom Host: header and curl fails to send a request on a re-used persistent connection and thus creates a new connection and resends it. It then sent two Host: headers. Cyrill's analysis was posted here: http://curl.haxx.se/mail/archive-2005-01/0022.html - Bruce Mitchener identified (bug report #1099640) the never-ending SOCKS5 problem with the version byte and the check for bad versions. Bruce has lots of clues on this, and based on his suggestion I've now removed the check of that byte since it seems to be able to contain 1 or 5. Daniel (10 January 2005) - Pavel Orehov reported memory problems with the multi interface in bug report #1098843. In short, a shared DNS cache was setup for a multi handle and when the shared cache was deleted before the individual easy handles, the latter cleanups caused read/writes to already freed memory. - Hzhijun reported a memory leak in the SSL certificate code, that leaked the remote certificate name when it didn't match the used host name. Gisle (8 January 2005) - Added Makefile.Watcom files (src/lib). Updated Makefile.dist. Daniel (7 January 2005) - Improved the test script's valgrind log parser to actually work! Also added the ability to disable the log scanner for specific test cases. Test case 509 results in numerous problems and leaks in OpenSSL and has to get it disabled. Daniel (6 January 2005) - Fixed a single-byte read out of bounds in test case 39 in the curl tool code (i.e not in the library). - Bug report #1097019 identified a problem when doing -d "data" with -G and sending it to two URLs with {}. Added test 199 to verify the fix. Daniel (4 January 2005) - Marty Kuhrt adjusted a VMS build script slightly - Kai Sommerfeld and Gisle Vanem fixed libcurl to build with IPv6 support on Win2000. Daniel (2 January 2005) - Alex Neblett updated the MSVC makefiles slightly.