Previously the host name buffer was only used if gethostname() exists,
but since we converted that into a curl private function that function
always exists and will be used so the buffer needs to exist for all
cases/systems.
A shared library tests/libtest/.libs/lihostname.so is preloaded in NTLM
test-cases to override the system implementation of gethostname(). It
makes it possible to test the NTLM authentication for exact match, and
this way test the implementation of MD4 and DES.
If LD_PRELOAD doesn't work, a debug build willl also workk as debug
builds are now made to prefer a specific environment variable and will
then return that content as host name instead of the actual one.
Kamil wrote the bulk of this, Daniel Stenberg polished it.
lib/Makefile.Watcom works fine already, for src/Makefile.Watcom we
need first to tweak src/Makefile.inc a bit - therefore the handtweaked
list still exists for now.
- make both libcurl and curl makefiles use register calling convention
(previously libcurl had stack calling convention).
- added include paths to the Watcom headers so its no longer required
to set the environment vars for this.
- added -wcd=201 to supress compiler warning about unreachable code.
- use macros for all tools, and removed dependency on GNU tools like rm.
- make ipv6 and debug builds controlable via env vars and so make them
optional instead of default.
- commented WINLDAPAPI and WINBERAPI since they broke with OW 1.8, and
it seems they're not needed (anymore?).
- added rule for hugehelp.c.cvs so that it will be created when not
already exist - this is required for building from a release tarball
since there we have no hugehelp.c.cvs, thus compilation broke.
- removed C_ARG creation from lib/Makefile.Watcom and use CFLAGS
directly as done too in src/Makefile.Watcom - this has the benefit
that we will see all active cflags and defines during compile.
- added LINK-ARG to src/Makefile.Watcom in order to better control
linker input.
- a couple of other minor makefile tweaks here and there ...
- added largefile support for Watcom builds to config-win32.h. Not yet
tested if it really works, but should since Win32 supports it.
- added loaddll stuff to speed up builds if supported.
Win64's 32 bit long but 64 bit size_t caused a warning that we avoid
with a typecast. A small whitespace indent fix was also applied.
Reported by: Adam Light
This passes -Werror to gcc when building curl and libcurl,
allowing easy dection of compile warnings.
Signed-off-by: Ben Greear <greearb@candelatech.com>
... since FTP is using it as well, and potentially other protocols!
Also, an #endif CURL_DISABLE_HTTP was incorrectly marked, as it seems to
end the proxy block instead.
The FTP implementation was missing a timestamp reset point, making the
waiting for responses after sending a post-transfer "QUOTE" command not
working as supposedly. This bug was introduced in 7.20.0
curl_multi perform has two phases: run through every easy handle calling
multi_runsingle and remove expired timers (timer removal).
If a small timer (e.g. 1-10ms) is set during multi_runsingle, then it's
possible that the timer has passed by when the timer removal runs. The
timer which was just added is then removed. This will potentially cause
the timer list to be empty and cause the next call to curl_multi_timeout
to return -1. Ideally, curl_multi_timeout should return 0 in this case.
One way to fix this is to move the struct timeval now = Curl_tvnow(); to
the top of curl_multi_perform. The change does that.
As mentioned in bug report #2956968, the HTTP code wouldn't send the
first empty chunk during the auth negotiation phase of the HTTP request
sending, so the server would wait for data to come and libcurl would
wait for data to arrive... I've made the code not enable chunked
encoding until the auth negotiation is done and thus this scenario
doesn't occur anymore.
Reported by: Sidney San Martn
Bug: http://curl.haxx.se/bug/view.cgi?id=2956968
When curl_multi_remove_handle() is called and an easy handle is returned
to the connection cache held in the multi handle, then we cannot allow
CURLINFO_LASTSOCKET to extract it since that will more or less encourage
that the user uses the socket while it can get used by libcurl again.
Without this fix, we'd get a segfault in Curl_getconnectinfo() trying to
dereference the NULL pointer in 'data->state.connc'.
Bug: http://curl.haxx.se/bug/view.cgi?id=3023840
When configured with '--without-ssl --with-nss', NTLM authentication
now uses NSS crypto library for MD5 and DES. For MD4 we have a local
implementation in that case. More details are available at
https://bugzilla.redhat.com/603783
In order to get it working, curl_global_init() must be called with
CURL_GLOBAL_SSL or CURL_GLOBAL_ALL. That's necessary because NSS needs
to be initialized globally and we do so only when the NSS library is
actually required by protocol. The mentioned call of curl_global_init()
is responsible for creating of the initialization mutex.
There was also slightly changed the NSS initialization scenario, in
particular, loading of the NSS PEM module. It used to be loaded always
right after the NSS library was initialized. Now the library is
initialized as soon as any SSL or NTLM is required, while the PEM module
is prevented from being loaded until the SSL is actually required.
When a hostname resolves to multiple IP addresses and the first one
tried doesn't work, the socket for the second attempt may get dropped on
the floor, causing the request to eventually time out. The issue is that
when using kqueue (as on mac and bsd platforms) instead of select, the
kernel removes the first fd from kqueue when it is closed (in trynextip,
connect.c:503). Trynextip() then goes on to open a new socket, which
gets assigned the same number as the one it just closed. Later in
multi.c, socket_cb is not called because the fd is already in
multi->sockhash, so the new socket is never added to kqueue.
The correct fix is to ensure that socket_cb is called to remove the fd
when trynextip() closes the socket, and again to re-add it after
singleipsocket(). I'm not sure how to cleanly do that, but the attached
patch works around the problem in an admittedly kludgy way by delaying
the close to ensure that the newly-opened socket gets a different fd.
Daniel's added comment: I didn't spot a way to easily do a nicer fix so
I've proceeded with Ben's patch.
Bug: http://curl.haxx.se/bug/view.cgi?id=3017819
Patch by: Ben Darnell
For example the libssh2 based functions return other negative
values than -1 to signal errors and it is important that we catch
them properly. Right before this, various failures from libssh2
were treated as negative download amounts which caused havoc.
My additional call to Curl_pgrsUpdate() would sometimes get
called even though there's no connection (left) so a NULL pointer
would get passed, causing a segfault.
1) no need to call the progress function twice when in the
CURLM_STATE_TOOFAST state.
2) Make sure that the progress callback's return code is
acknowledged when used
As long as no error is reported, the progress function can get
called. This may be a little TOO often so we should keep an eye
on this and possibly make this conditional somehow.
Older unixes want an 'int' instead of 'size_t' as the 3rd
argumment so before this change it would cause warnings such as:
There is an implicit conversion from "unsigned long" to "int";
rounding, sign extension, or loss of accuracy may result.
Was seeing spurious SSL connection aborts using libcurl and
OpenSSL. I tracked it down to uncleared error state on the
OpenSSL error stack - patch attached deals with that.
Rough idea of problem:
Code that uses libcurl calls some library that uses OpenSSL but
don't clear the OpenSSL error stack after an error.
ssluse.c calls SSL_read which eventually gets an EWOULDBLOCK from
the OS. Returns -1 to indicate an error
ssluse.c calls SSL_get_error. First thing, SSL_get_error calls
ERR_get_error to check the OpenSSL error stack, finds an old
error and returns SSL_ERROR_SSL instead of SSL_ERROR_WANT_READ or
SSL_ERROR_WANT_WRITE.
ssluse.c returns an error and aborts the connection
Solution:
Clear the openssl error stack before calling SSL_* operation if
we're going to call SSL_get_error afterwards.
Notes:
This is much more likely to happen with multi because it's easier
to intersperse other calls to the OpenSSL library in the same
thread.
Enable OpenLDAP support for cygwin builds. This support was disabled back
in 2008 due to incompatibilities between OpenSSL and OpenLDAP headers.
cygwin's OpenSSL 0.9.8l and OpenLDAP 2.3.43 versions on cygwin 1.5.25
allow building an OpenLDAP enabled libcurl supporting back to Windows 95.
Remove non-functional CURL_LDAP_HYBRID code and references.
Jason McDonald posted bug report #3006786 when he found that the
SFTP code didn't timeout properly in several places in the code
even if a timeout was set properly.
Based on his suggested patch, I wrote a different implementation
that I think addressed the issue better and also uses the connect
timeout for the initial part of the SSH/SFTP done during the
"protocol connect" phase.
(http://curl.haxx.se/bug/view.cgi?id=3006786)
Igor Novoseltsev reported a problem with the multi socket API and
using timeouts and timers. It boiled down to a problem with
libcurl's use of GetTickCount() interally to figure out the
current time, while Igor's own application code used another
function call.
It made his app call the socket API timeout function a bit
_before_ libcurl would consider the timeout to trigger, and that
could easily lead to timeouts or stalls in the app. It seems
GetTickCount() in general often has no better resolution than
16ms and switching to the alternative function
QueryPerformanceCounter has its share of problems:
http://www.virtualdub.org/blog/pivot/entry.php?id=106
We address this problem by simply having libcurl treat timers
that already has occured or will occur within 40ms subject for
treatment. I'm confident that there are other implementations and
operating systems with similarly in accurate timer functions so
it makes sense to have applied generically and I don't believe we
sacrifice much by adding a 40ms inaccuracy on these timeouts.
makes the LDAP code much cleaner, nicer and in general being a
better libcurl citizen. If a new enough OpenLDAP version is
detect, the new and shiny lib/openldap.c code is then used
instead of the old cruft
Code by Howard, minor cleanups by Daniel.
bool in curl internals is unsigned char and should not be used
to receive return value from functions returning int - this fails
when using IBM VisualAge and Tru64 compilers.
Eric Mertens posted bug #3003705: when we made TFTP use the
correct timeout option when sent to the server (fixed May 18th
2010) it became obvious that libcurl used invalid timeout values
(300 by default while the RFC allows nothing above 255). While of
course it is obvious that as TFTP has worked thus far without
being able to set timeout at all, just removing the setting
wouldn't make any difference in behavior. I decided to still keep
it (but fix the problem) as it now actually allows for easier
(future) customization of the timeout.
(http://curl.haxx.se/bug/view.cgi?id=3003705)
In a normal expression, doing [unsigned short] + 1 will not wrap
at 16 bits so the comparisons and outputs were done wrong. I
added a macro do make sure it gets done right.
Douglas Kilpatrick filed bug report #3004787 about it:
http://curl.haxx.se/bug/view.cgi?id=3004787
By undefing a bunch of E* defines that VC10 has started to define
but that we redefine internally to their WSA* alternatives when
building for Windows.
Eric Mertens posted bug report #3003005 pointing out that the
libcurl TFTP code was not sending the timeout option properly to
the server, and suggested a fix.
(http://curl.haxx.se/bug/view.cgi?id=3003005)
John-Mark Bell filed bug #3000052 that identified a problem (with
an associated patch) with the OpenSSL handshake state machine
when the multi interface is used:
Performing an https request using a curl multi handle and using
select or epoll to wait for events results in a hang. It appears
that the cause is the fix for bug #2958179, which makes
ossl_connect_common unconditionally return from the step 2 loop
when fetching from a multi handle.
When ossl_connect_step2 has completed, it updates
connssl->connecting_state to ssl_connect_3. ossl_connect_common
will then return to the caller, as a multi handle is in
use. Eventually, the client code will call curl_multi_fdset to
obtain an updated fdset to select or epoll on. For https
requests, curl_multi_fdset will cause https_getsock to be called.
https_getsock will only return a socket handle if the
connecting_state is ssl_connect_2_reading or
ssl_connect_2_writing. Therefore, the client will never obtain a
valid fdset, and thus not drive the multi handle, resulting in a
hang.
(http://curl.haxx.se/bug/view.cgi?id=3000052)
Sebastian V reported bug #3000056 identifying a problem with
redirect following. It showed that when curl followed redirects
it didn't properly ignore the response body of the 30X response
if that response was using compressed Content-Encoding!
(http://curl.haxx.se/bug/view.cgi?id=3000056)
"The BSD version of PolarSSL was made for migratory purposes only and is not
maintained. The GPL version of PolarSSL is actually the only actively
developed version, so I would be very reluctant to use the BSD version." /
Paul Bakker, PolarSSL hacker.
Signed-off-by: Hoi-Ho Chan <hoiho.chan@gmail.com>
FTP(S) use two connections that can be set to different recv and
send functions independently, so by introducing recv+send pairs
in the same manner we already have sockets/connections we can
work with FTPS fine.
This commit fixes the FTPS regression introduced in change d64bd82.
Dirk Manske reported a regression. When connecting with the multi
interface, there were situations where libcurl wouldn't store
connect time correctly as it used to (and is documented to) do.
Using his fine sample program we could repeat it, and I wrote up
test case 573 using that code. The problem does not easily show
itself using the local test suite though.
The fix, also as suggested by Dirk, is a bit on the ugly side as
it adds yet another call to Curl_verboseconnect() and setting the
TIMER_CONNECT time. That situation is subject for some closer
inspection in the future.
Howard Chu brought the bulk work of this patch that properly
moves out the sending and recving of data to the parts of the
code that are properly responsible for the various ways of doing
so.
Daniel Stenberg assisted with polishing a few bits and fixed some
minor flaws in the original patch.
Another upside of this patch is that we now abuse CURLcodes less
with the "magic" -1 return codes and instead use CURLE_AGAIN more
consistently.
This is Hoi-Ho Chan's patch with some minor fixes by me. There
are some potential issues in this, but none worse than we can
sort out on the list and over time.
The main change is to allow input from user-specified methods,
when they are specified with CURLOPT_READFUNCTION.
All calls to fflush(stdout) in telnet.c were removed, which makes
using 'curl telnet://foo.com' painful since prompts and other data
are not always returned to the user promptly. Use
'curl --no-buffer telnet://foo.com' instead. In general,
the user should have their CURLOPT_WRITEFUNCTION do a fflush
for interactive use.
Also fix assumption that reading from stdin never returns < 0.
Old code could crash in that case.
Call progress functions in telnet main loop.
Signed-off-by: Ben Greear <greearb@candelatech.com>
Make sure we don't call memcpy() if the argument is NULL even
though we also passed a zero length then, as the clang analyzer
whined and we want to limit warnings (even false positives) when
they're this easy to fix.
The change of (char) to (unsigned char) will fix long user names
and passwords on systems that have the char type signed by
default.
The recent overhaul of the SSL recv function made this treat a
zero returned from gnutls_record_recv() as an error, and this
caused our HTTPS test cases to fail. We leave it to upper layer
code to detect if an EOF is a problem or not.
This code would previously use dns_entry->addr->ai_canonname
instead of the given host name, which caused us grief and
problems since not all our resolver options do the reverse lookup
and I would also guess that it caused problems with KRB5/GSS with
virtual name-based hosts. Now the host name from the URL is used.
As reported in bug report #2987196, the code for ipv6 already did
the setting of this bit correctly so we copied that logic into
the Curl_ipv4_resolve_r() function as well. KRB code is the only
code we know that might need the cannonical name so only resolve
it for such requests!
Prefixing the FTP quote commands with an asterisk really only
worked for the postquote actions. This is now fixed and test case
227 has been extended to verify.
Matt Wixson found and fixed a bug in the SCP/SFTP area where the
code treated a 0 return code from libssh2 to be the same as
EAGAIN while in reality it isn't. The problem caused a hang in
SFTP transfers from a MessageWay server.
strlen() returns size_t, but ssh libraries are wanting 'unsigned int'. Add
explicit casts and use _ex versions of the ssh library calls.
Signed-off-by: Ben Greear <greearb@candelatech.com>
If you pass a URL to pop3 that does not contain a message ID as
part of the URL, it will currently ask for 'INBOX' which just
causes the pop3 server to return an error.
The change makes libcurl treat en empty message ID as a request
for LIST (list of pop3 message IDs). User's code could then
parse this and download individual messages as desired.
Ben Greear brought a patch that from now on allows all protocols
to specify name and user within the URL, in the same manner HTTP
and FTP have been allowed to in the past - although far from all
of the libcurl supported protocols actually have that feature in
their URL definition spec.
Bob Richmond: There's an annoying situation where libcurl will
read new HTTP response data from a socket, then check if it's a
timeout if one is set. If the last packet received constitutes
the end of the response body, libcurl still treats it as a
timeout condition and reports a message like:
"Operation timed out after 3000 milliseconds with 876 out of 876
bytes received"
It should only a timeout if the timer lapsed and we DIDN'T
receive the end of the response body yet.
This commit fixes the cmake build of curl, and cleans up the
cmake code a little. It removes some commented out code and
some trailing whitespace. To get curl to build the binary
tree include/curl directory needed to be added to the include
path. Also, SIZEOF_SHORT needed to be added. A check for the
lack of defines of SIZEOF_* for warnless.c was added.
Kenny To filed the bug report #2963679 with patch to fix a
problem he experienced with doing multi interface HTTP POST over
a proxy using PROXYTUNNEL. He found a case where it would connect
fine but bits.tcpconnect was not set correct so libcurl didn't
work properly.
(http://curl.haxx.se/bug/view.cgi?id=2963679)
Akos Pasztory filed debian bug report #572276http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=572276
mentioning a problem with a resource that returns chunked-encoded
_and_ with a Content-Length and libcurl failed to properly ignore
the latter information.
Hauke Duden provided an example program that made the multi
interface crash. His example simply used the multi interface and
did first one FTP transfer and after completion it used a second
easy handle and did another FTP transfer on the same FTP server.
This triggered a bug in the "delayed easy handle kill" system
that curl uses: when an FTP connection is left alive it must keep
an easy handle around internally - only for the purpose of having
an easy handle when it later disconnects it. The code assumed
that when the easy handle was removed and an internal reference
was made, that version could be killed later on when a new easy
handle came using the same connection. This was wrong as Hauke's
example showed that the removed handle wasn't killed for real
until later. This caused a double close attempt => segfault.
Looking at the code of Curl_resolv_timeout() in hostip.c, I think
that in case of a timeout, the signal handler for SIGALRM never
gets removed. I think that in my case it gets executed at some
point later on when execution has long left Curl_resolv_timeout()
or even the cURL library.
The code that is jumped to with siglongjmp() simply sets the
error message to "name lookup timed out" and then returns with
CURLRESOLV_ERROR. I guess that instead of simply returning
without cleaning up, the code should have a goto that jumps to
the spot right after the call to Curl_resolv().
Error codes were not properly returned to the main curl code (and on to apps
using libcurl).
tftp was crapping out when tsize == 0 on upload, but I see no reason to fail
to upload just because the remote file is zero-length. Ignore tsize option on
upload.
The problem mentioned on Dec 10 2009
(http://curl.haxx.se/bug/view.cgi?id=2905220) was only partially fixed.
Partially because an easy handle can be associated with many connections in
the cache (e.g. if there is a redirect during the lifetime of the easy
handle). The previous patch only cleaned up the first one. The new fix now
removes the easy handle from all connections, not just the first one.
makes sure that when using sub-second timeouts, there's no final bad 1000ms
wait. Previously, a sub-second timeout would often make the elapsed time end
up the time rounded up to the nearest second (e.g. 1s for 200ms timeout)
the global timeout if set. Also, as was reported in the bug report #2956437
by Ryan Chan, the time stamp to use as basis for the per command timeout was
not set properly in the DONE phase for FTP (and not for SMTP) so I fixed
that just now. This was a regression compared to 7.19.7 due to the
conversion of FTP code over to the generic pingpong concepts.
http://curl.haxx.se/bug/view.cgi?id=2956437
- SMTP falls back to RFC821 HELO when EHLO fails (and SSL is not required).
- Use of true local host name (i.e.: via gethostname()) when available, as default argument to SMTP HELO/EHLO.
- Test case 804 for HELO fallback.
properly in angle brackets. Recipients provided with CURLOPT_MAIL_RCPT now
get angle bracket wrapping automatically by libcurl unless the recipient
starts with an angle bracket as then the app is assumed to deal with that
properly on its own.
full DATA has been sent, and I modified the test SMTP server to also send
that response. As usual, the DONE operation that is made after a completed
transfer is still not doable in a non-blocking way so this waiting for 250
is unfortunately made blockingly.
in the same RCPT TO line, when they should be sent in separate single
commands. I updated test case 802 to verify this.
- I also fixed a bad use of my_setopt_str() of CURLOPT_MAIL_RCPT in the curl
tool which made it try to output it as string for the --libcurl feature
which could lead to crashes.
to automatically uncompress it with the CURLOPT_ENCODING option, libcurl
could wrongly provide the callback with more data than what the maximum
documented amount. An application could thus get tricked into badness if the
maximum limit was trusted to be enforced by libcurl itself (as it is
documented).
This is further detailed and explained in the libcurl security advisory
20100209 at
http://curl.haxx.se/docs/adv_20100209.html
simply check for CURLM_CALL_MULTI_PERFORM internally. This has the added
benefit that this goes in line with my long-term wishes to get rid of the
CURLM_CALL_MULTI_PERFORM all together from the public API.
from hostip.h to setup.h in order to allow proper inclusion in any file.
This represents no functional change at all in which resolver is used,
everything still works as usual, internally and externally there is no
difference in behavior.
HTTP Cookie: header _needs_ to be sorted on the path length in the cases
where two cookies using the same name are set more than once using
(overlapping) paths. Realizing this, identically named cookies must be
sorted correctly. But detecting only identically named cookies and take care
of them individually is harder than just to blindly and unconditionally sort
all cookies based on their path lengths. All major browsers also already do
this, so this makes our behavior one step closer to them in the cookie area.
Test case 8 was the only one that broke due to this change and I updated it
accordingly.
again when downloading files over FTP using ASCII and it turns out that the
final size of the file is not the same as the initial size the server
reported. This is very common since servers don't take the newline
conversions into account.
much as possible in one go, as long as it doesn't block and hasn't reached the
end of the state machine.
This avoids spurious -1 returns from curl_multi_fdset() simply because
previously it would return from this function without anything in EWOUDLBLOCK
and thus basically it wasn't actually waiting for anything!!
state, we return CURLM_CALL_MULTI_PERFORM unconditionally then so that we
can act faster like in the case the protocol-specific connect doesn't block
on anything and we can just persue on the next action immediately. It also
then avoids a case where curl_multi_fdset() would return -1.
ossl_connect_step3() increments an SSL session handle reference counter on
each call. When sessions are re-used this reference counter may be
incremented many times, but it will be decremented only once when done (by
Curl_ossl_session_free()); and the internal OpenSSL data will not be freed
if this reference count remains positive. When a session is re-used the
reference counter should be corrected by explicitly calling
SSL_SESSION_free() after each consecutive SSL_get1_session() to avoid
introducing a memory leak.
(http://curl.haxx.se/bug/view.cgi?id=2926284)
command is a special "hack" used by the drftpd server, but even though it is
a custom extension I've deemed it fine to add to libcurl since this server
seems to survive and people keep using it and want libcurl to support
it. The new libcurl option is named CURLOPT_FTP_USE_PRET, and it is also
usable from the curl tool with --ftp-pret. Using this option on a server
that doesn't support this command will make libcurl fail.
sequences in uploaded data. The test server doesn't "decode" escaped dot-lines
but instead test cases must be written to take them into account. Added test
case 803 to verify dot-escaping.
detects and uses proxies based on the environment variables. If the proxy
was given as an explicit option it worked, but due to the setup order
mistake proxies would not be used fine for a few protocols when picked up
from '[protocol]_proxy'. Obviously this broke after 7.19.4. I now also added
test case 1106 that verifies this functionality.
(http://curl.haxx.se/bug/view.cgi?id=2913886)
