This commit is several drafts squashed together. The changes from each
draft are noted below. If any changes are similar and possibly
contradictory the change in the latest draft takes precedence.
Bug: https://github.com/bagder/curl/issues/244
Reported-by: Chris Araman
%%
%% Draft 1
%%
- return 0 if len == 0. that will have to be documented.
- continue on and process the caches regardless of raw recv
- if decrypted data will be returned then set the error code to CURLE_OK
and return its count
- if decrypted data will not be returned and the connection has closed
(eg nread == 0) then return 0 and CURLE_OK
- if decrypted data will not be returned and the connection *hasn't*
closed then set the error code to CURLE_AGAIN --only if an error code
isn't already set-- and return -1
- narrow the Win2k workaround to only Win2k
%%
%% Draft 2
%%
- Trying out a change in flow to handle corner cases.
%%
%% Draft 3
%%
- Back out the lazier decryption change made in draft2.
%%
%% Draft 4
%%
- Some formatting and branching changes
- Decrypt all encrypted cached data when len == 0
- Save connection closed state
- Change special Win2k check to use connection closed state
%%
%% Draft 5
%%
- Default to CURLE_AGAIN in cleanup if an error code wasn't set and the
connection isn't closed.
%%
%% Draft 6
%%
- Save the last error only if it is an unrecoverable error.
Prior to this I saved the last error state in all cases; unfortunately
the logic to cover that in all cases would lead to some muddle and I'm
concerned that could then lead to a bug in the future so I've replaced
it by only recording an unrecoverable error and that state will persist.
- Do not recurse on renegotiation.
Instead we'll continue on to process any trailing encrypted data
received during the renegotiation only.
- Move the err checks in cleanup after the check for decrypted data.
In either case decrypted data is always returned but I think it's easier
to understand when those err checks come after the decrypted data check.
%%
%% Draft 7
%%
- Regardless of len value go directly to cleanup if there is an
unrecoverable error or a close_notify was already received. Prior to
this change we only acknowledged those two states if len != 0.
- Fix a bug in connection closed behavior: Set the error state in the
cleanup, because we don't know for sure it's an error until that time.
- (Related to above) In the case the connection is closed go "greedy"
with the decryption to make sure all remaining encrypted data has been
decrypted even if it is not needed at that time by the caller. This is
necessary because we can only tell if the connection closed gracefully
(close_notify) once all encrypted data has been decrypted.
- Do not renegotiate when an unrecoverable error is pending.
%%
%% Draft 8
%%
- Don't show 'server closed the connection' info message twice.
- Show an info message if server closed abruptly (missing close_notify).
With many easy handles using the same connection for multiplexing, it is
important we store and keep the transfer-oriented stuff in the
SessionHandle so that callbacks and callback data work fine even when
many easy handles share the same physical connection.
SSLeay was the name of the library that was subsequently turned into
OpenSSL many moons ago (1999). curl does not work with the old SSLeay
library since years. This is now reflected by only using USE_OPENSSL in
code that depends on OpenSSL.
This option can be used to enable/disable certificate status verification using
the "Certificate Status Request" TLS extension defined in RFC6066 section 8.
This also adds the CURLE_SSL_INVALIDCERTSTATUS error, to be used when the
certificate status verification fails, and the Curl_ssl_cert_status_request()
function, used to check whether the SSL backend supports the status_request
extension.
There was a confusion between these: this commit tries to disambiguate them.
- Scope can be computed from the address itself.
- Scope id is scope dependent: it is currently defined as 1-based local
interface index for link-local scoped addresses, and as a site index(?) for
(obsolete) site-local addresses. Linux only supports it for link-local
addresses.
The URL parser properly parses a scope id as an interface index, but stores it
in a field named "scope": confusion. The field has been renamed into "scope_id".
Curl_if2ip() used the scope id as it was a scope. This caused failures
to bind to an interface.
Scope is now computed from the addresses and Curl_if2ip() matches them.
If redundantly specified in the URL, scope id is check for mismatch with
the interface index.
This commit should fix SF bug #1451.
The ability to do HTTP requests over a UNIX domain socket has been
requested before, in Apr 2008 [0][1] and Sep 2010 [2]. While a
discussion happened, no patch seems to get through. I decided to give it
a go since I need to test a nginx HTTP server which listens on a UNIX
domain socket.
One patch [3] seems to make it possible to use the
CURLOPT_OPENSOCKETFUNCTION function to gain a UNIX domain socket.
Another person wrote a Go program which can do HTTP over a UNIX socket
for Docker[4] which uses a special URL scheme (though the name contains
cURL, it has no relation to the cURL library).
This patch considers support for UNIX domain sockets at the same level
as HTTP proxies / IPv6, it acts as an intermediate socket provider and
not as a separate protocol. Since this feature affects network
operations, a new feature flag was added ("unix-sockets") with a
corresponding CURL_VERSION_UNIX_SOCKETS macro.
A new CURLOPT_UNIX_SOCKET_PATH option is added and documented. This
option enables UNIX domain sockets support for all requests on the
handle (replacing IP sockets and skipping proxies).
A new configure option (--enable-unix-sockets) and CMake option
(ENABLE_UNIX_SOCKETS) can disable this optional feature. Note that I
deliberately did not mark this feature as advanced, this is a
feature/component that should easily be available.
[0]: http://curl.haxx.se/mail/lib-2008-04/0279.html
[1]: http://daniel.haxx.se/blog/2008/04/14/http-over-unix-domain-sockets/
[2]: http://sourceforge.net/p/curl/feature-requests/53/
[3]: http://curl.haxx.se/mail/lib-2008-04/0361.html
[4]: https://github.com/Soulou/curl-unix-socket
Signed-off-by: Peter Wu <peter@lekensteyn.nl>
Reworked the input token (challenge message) storage as what is passed
to the buf and desc in the response generation are typically blobs of
data rather than strings, so this is more in keeping with other areas
of the SSPI code, such as the NTLM message functions.
When duplicating a handle, the data to post was duplicated using
strdup() when it could be binary and contain zeroes and it was not even
zero terminated! This caused read out of bounds crashes/segfaults.
Since the lib/strdup.c file no longer is easily shared with the curl
tool with this change, it now uses its own version instead.
Bug: http://curl.haxx.se/docs/adv_20141105.html
CVE: CVE-2014-3707
Reported-By: Symeon Paraschoudis
Typically the USE_WINDOWS_SSPI definition would not be used when the
CURL_DISABLE_CRYPTO_AUTH define is, however, it is still a valid build
configuration and, as such, the SASL Kerberos V5 (GSSAPI) authentication
data structures and functions would incorrectly be used when they
shouldn't be.
Introduced a new USE_KRB5 definition that takes into account the use of
CURL_DISABLE_CRYPTO_AUTH like USE_SPNEGO and USE_NTLM do.
Option --pinnedpubkey takes a path to a public key in DER format and
only connect if it matches (currently only implemented with OpenSSL).
Provides CURLOPT_PINNEDPUBLICKEY for curl_easy_setopt().
Extract a public RSA key from a website like so:
openssl s_client -connect google.com:443 2>&1 < /dev/null | \
sed -n '/-----BEGIN/,/-----END/p' | openssl x509 -noout -pubkey \
| openssl rsa -pubin -outform DER > google.com.der
Updated to use a dynamic buffer for the SPN generation via the recently
introduced Curl_sasl_build_spn() function rather than a fixed buffer of
1024 characters, which should have been more than enough, but by using
the new function removes the need for another variable sname to do the
wide character conversion in Unicode builds.
Given the SSPI package info query indicates a token size of 2888 bytes,
and as with the Winbind code and commit 9008f3d56, use a dynamic buffer
for the Type-1 and Type-3 message generation rather than a fixed buffer
of 1024 bytes.
- Replace CURLAUTH_GSSNEGOTIATE with CURLAUTH_NEGOTIATE
- CURL_VERSION_GSSNEGOTIATE is deprecated which
is served by CURL_VERSION_SSPI, CURL_VERSION_GSSAPI and
CURUL_VERSION_SPNEGO now.
- Remove display of feature 'GSS-Negotiate'
Make all code use connclose() and connkeep() when changing the "close
state" for a connection. These two macros take a string argument with an
explanation, and debug builds of curl will include that in the debug
output. Helps tracking connection re-use/close issues.
In commit 0b3750b5c2 (released in 7.36.0) we fixed a timeout issue
but instead broke the timings.
To fix this, I introduce a new timestamp to use for the timeouts and
restored the previous timestamp and timestamp position so that the old
timer functionality is restored.
In addition to that, that change also broke connection timeouts for when
more than one connect was used (as it would then count the total time
from the first connect and not for the most recent one). Now
Curl_timeleft() has been modified so that it checks against different
start times depending on which timeout it checks.
Test 1303 is updated accordingly.
Bug: http://curl.haxx.se/mail/lib-2014-05/0147.html
Reported-by: Ryan Braud
set.infilesize in this case was modified in several places, which could
lead to repeated requests using the same handle to get unintendent/wrong
consequences based on what the previous request did!
This makes the findprotocol() function work as intended so that libcurl
can properly be restricted to not support HTTP while still supporting
HTTPS - since the HTTPS handler previously set both the HTTP and HTTPS
bits in the protocol field.
This fixes --proto and --proto-redir for most SSL protocols.
This is done by adding a few new convenience defines that groups HTTP
and HTTPS, FTP and FTPS etc that should then be used when the code wants
to check for both protocols at once. PROTO_FAMILY_[protocol] style.
Bug: https://github.com/bagder/curl/pull/97
Reported-by: drizzt
In addition to FTP, other connection based protocols such as IMAP, POP3,
SMTP, SCP, SFTP and LDAP require a new connection when different log-in
credentials are specified. Fixed the detection logic to include these
other protocols.
Bug: http://curl.haxx.se/docs/adv_20140326A.html
Rename x509_cert to x509_crt and add "compat-1.2.h"
include.
This would still need some more thorough conversion
in order to drop "compat-1.2.h" include.
Port number zero is perfectly allowed to connect to. I moved to storing
the remote port number in an int so that -1 means undefined and 0-65535
can be used for legitimate port numbers.
when using --http2 one can now selectively disable NPN or ALPN with
--no-alpn and --no-npn. for now honored with NSS only.
TODO: honor this option with GnuTLS and OpenSSL
1) Renamed curl_tlsinfo to curl_tlssessioninfo as discussed on the
mailing list.
2) Renamed curl_ssl_backend to curl_sslbackend so it doesn't follow our
function naming convention.
3) Updated sessioninfo.c example accordingly.
Added new API for returning a SSL backend type and pointer, in order to
allow access to the TLS internals, that may then be used to obtain X509
certificate information for example.
This patch invokes two socket connect()s nearly simultaneously, and
the socket that is first connected "wins" and is subsequently used for
the connection. The other is terminated.
There is a very slight IPv4 preference, in that if both sockets connect
simultaneously IPv4 is checked first and thus will win.
Added the ability to specify an XOAUTH2 bearer token [RFC6750] via the
option CURLOPT_XOAUTH2_BEARER for authentication using RFC6749 "OAuth
2.0 Authorization Framework".
We've announced this pending removal for a long time and we've
repeatedly asked if anyone would care or if anyone objects. Nobody has
objected. It has probably not even been working for a good while since
nobody has tested/used this code recently.
The stuff in krb4.h that was generic enough to be used by other sources
is now present in security.h
libcurl quietly truncates usernames, passwords, and options from
before an '@' sign in a URL to 255 (= MAX_CURL_PASSWORD_LENGTH - 1)
characters to fit in fixed-size buffers on the stack. Allocate a
buffer large enough to fit the parsed fields on the fly instead to
support longer passwords.
After this change, there are no more uses of MAX_CURL_OPTIONS_LENGTH
left, so stop defining that constant while at it. The hardcoded max
username and password length constants, on the other hand, are still
used in HTTP proxy credential handling (which this patch doesn't
touch).
Reported-by: Colby Ranger
All protocol handler structs are now opaque (void *) in the
SessionHandle struct and moved in the request-specific sub-struct
'SingleRequest'. The intension is to keep the protocol specific
knowledge in their own dedicated source files [protocol].c etc.
There's some "leakage" where this policy is violated, to be addressed at
a later point in time.
1 - always allocate the struct in protocol->setup_connection. Some
protocol handlers had to get this function added.
2 - always free at the end of a request. This is also an attempt to keep
less memory in the handle after it is completed.
This is a regression as this logic used to work. It isn't clear when it
broke, but I'm assuming in 7.28.0 when we went all-multi internally.
This likely never worked with the multi interface. As the failed
connection is detected once the multi state has reached DO_MORE, the
Curl_do_more() function was now expanded somewhat so that the
ftp_do_more() function can request to go "back" to the previous state
when it makes another attempt - using PASV.
Added test case 1233 to verify this fix. It has the little issue that it
assumes no service is listening/accepting connections on port 1...
Reported-by: byte_bucket in the #curl IRC channel
The motivation for having a separate struct that keep track of an easy
handle when using the multi handle was removed when we switched to
always using the multi interface internally. Now they were just two
separate struct that was always allocated for each easy handle.
This first step just moves the Curl_one_easy struct members into the
SessionHandle struct and hides this somehow (== keeps the source code
changes to a minimum) by defining Curl_one_easy to SessionHandle
The biggest changes in this commit are:
1 - the linked list of easy handles had to be changed somewhat due
to the new struct layout. This made the main linked list pointer
get renamed to 'easyp' and there's also a new pointer to the last
node, called easylp. It is no longer circular but ends with ->next
pointing to NULL. New nodes are still added last.
2 - easy->state is now called easy->mstate to avoid name collision
CURLOPT_XFERINFOFUNCTION is now the preferred progress callback function
and CURLOPT_PROGRESSFUNCTION is considered deprecated.
This new callback uses pure 'curl_off_t' arguments to pass on full
resolution sizes. It otherwise retains the same characteristics: the
same call rate, the same meanings for the arguments and the return code
is used the same way.
The progressfunc.c example is updated to show how to use the new
callback for newer libcurls while supporting the older one if built with
an older libcurl or even built with a newer libcurl while running with
an older.
In addition to parsing the optional login options from the URL, added
support for parsing them from CURLOPT_USERPWD, to allow the following
supported command line:
--user username:password;options
As well as parsing the username and password from the URL, added support
for parsing the optional options part from the login details, to allow
the following supported URL format:
schema://username:password;options@example.com/path?q=foobar
This will only be used by IMAP, POP3 and SMTP at present but any
protocol that may be given login options in the URL will be able to
add support for them.
At some point recently we lost the default value for the easy handle's
connection cache, and this change puts it back to 5 - which is the
former default value and it is documented in the curl_easy_setopt.3 man
page.
Introducing a number of options to the multi interface that
allows for multiple pipelines to the same host, in order to
optimize the balance between the penalty for opening new
connections and the potential pipelining latency.
Two new options for limiting the number of connections:
CURLMOPT_MAX_HOST_CONNECTIONS - Limits the number of running connections
to the same host. When adding a handle that exceeds this limit,
that handle will be put in a pending state until another handle is
finished, so we can reuse the connection.
CURLMOPT_MAX_TOTAL_CONNECTIONS - Limits the number of connections in total.
When adding a handle that exceeds this limit,
that handle will be put in a pending state until another handle is
finished. The free connection will then be reused, if possible, or
closed if the pending handle can't reuse it.
Several new options for pipelining:
CURLMOPT_MAX_PIPELINE_LENGTH - Limits the pipeling length. If a
pipeline is "full" when a connection is to be reused, a new connection
will be opened if the CURLMOPT_MAX_xxx_CONNECTIONS limits allow it.
If not, the handle will be put in a pending state until a connection is
ready (either free or a pipe got shorter).
CURLMOPT_CONTENT_LENGTH_PENALTY_SIZE - A pipelined connection will not
be reused if it is currently processing a transfer with a content
length that is larger than this.
CURLMOPT_CHUNK_LENGTH_PENALTY_SIZE - A pipelined connection will not
be reused if it is currently processing a chunk larger than this.
CURLMOPT_PIPELINING_SITE_BL - A blacklist of hosts that don't allow
pipelining.
CURLMOPT_PIPELINING_SERVER_BL - A blacklist of server types that don't allow
pipelining.
See the curl_multi_setopt() man page for details.
Created a new IMAP structure and changed the type of the imap proto
variable in connectdata from FTP* to the new IMAP*.
Moved the mailbox variable from the per-connection struct imap_conn to
the new per-request struct and fixed references accordingly.
An ambiguity in the SSLWrite() documentation lead to a bad inference in the
code where we assumed SSLWrite() returned the amount of bytes written to
the socket, when that is not actually true; it returns the amount of data
that is buffered for writing to the socket if it returns errSSLWouldBlock.
Now darwinssl_send() returns CURLE_AGAIN if data is buffered but not written.
Reference URL: http://curl.haxx.se/mail/lib-2013-02/0145.html
Remove internal separated behavior of the easy vs multi intercace.
curl_easy_perform() is now using the multi interface itself.
Several minor multi interface quirks and bugs have been fixed in the
process.
Much help with debugging this has been provided by: Yang Tse
This commit renames lib/setup.h to lib/curl_setup.h and
renames lib/setup_once.h to lib/curl_setup_once.h.
Removes the need and usage of a header inclusion guard foreign
to libcurl. [1]
Removes the need and presence of an alarming notice we carried
in old setup_once.h [2]
----------------------------------------
1 - lib/setup_once.h used __SETUP_ONCE_H macro as header inclusion guard
up to commit ec691ca3 which changed this to HEADER_CURL_SETUP_ONCE_H,
this single inclusion guard is enough to ensure that inclusion of
lib/setup_once.h done from lib/setup.h is only done once.
Additionally lib/setup.h has always used __SETUP_ONCE_H macro to
protect inclusion of setup_once.h even after commit ec691ca3, this
was to avoid a circular header inclusion triggered when building a
c-ares enabled version with c-ares sources available which also has
a setup_once.h header. Commit ec691ca3 exposes the real nature of
__SETUP_ONCE_H usage in lib/setup.h, it is a header inclusion guard
foreign to libcurl belonging to c-ares's setup_once.h
The renaming this commit does, fixes the circular header inclusion,
and as such removes the need and usage of a header inclusion guard
foreign to libcurl. Macro __SETUP_ONCE_H no longer used in libcurl.
2 - Due to the circular interdependency of old lib/setup_once.h and the
c-ares setup_once.h header, old file lib/setup_once.h has carried
back from 2006 up to now days an alarming and prominent notice about
the need of keeping libcurl's and c-ares's setup_once.h in sync.
Given that this commit fixes the circular interdependency, the need
and presence of mentioned notice is removed.
All mentioned interdependencies come back from now old days when
the c-ares project lived inside a curl subdirectory. This commit
removes last traces of such fact.
This reverts renaming and usage of lib/*.h header files done
28-12-2012, reverting 2 commits:
f871de0... build: make use of 76 lib/*.h renamed files
ffd8e12... build: rename 76 lib/*.h files
This also reverts removal of redundant include guard (redundant thanks
to changes in above commits) done 2-12-2013, reverting 1 commit:
c087374... curl_setup.h: remove redundant include guard
This also reverts renaming and usage of lib/*.c source files done
3-12-2013, reverting 3 commits:
13606bb... build: make use of 93 lib/*.c renamed files
5b6e792... build: rename 93 lib/*.c files
7d83dff... build: commit 13606bbfde follow-up 1
Start of related discussion thread:
http://curl.haxx.se/mail/lib-2013-01/0012.html
Asking for confirmation on pushing this revertion commit:
http://curl.haxx.se/mail/lib-2013-01/0048.html
Confirmation summary:
http://curl.haxx.se/mail/lib-2013-01/0079.html
NOTICE: The list of 2 files that have been modified by other
intermixed commits, while renamed, and also by at least one
of the 6 commits this one reverts follows below. These 2 files
will exhibit a hole in history unless git's '--follow' option
is used when viewing logs.
lib/curl_imap.h
lib/curl_smtp.h
A bundle is a list of all persistent connections to the same host.
The connection cache consists of a hash of bundles, with the
hostname as the key.
The benefits may not be obvious, but they are two:
1) Faster search for connections to reuse, since the hash
lookup only finds connections to the host in question.
2) It lays out the groundworks for an upcoming patch,
which will introduce multiple HTTP pipelines.
This patch also removes the awkward list of "closure handles",
which were needed to send QUIT commands to the FTP server
when closing a connection.
Now we allocate a separate closure handle and use that
one to close all connections.
This has been tested in a live system for a few weeks, and of
course passes the test suite.
After a research team wrote a document[1] that found several live source
codes out there in the wild that misused the CURLOPT_SSL_VERIFYHOST
option thinking it was a boolean, this change now bans 1 as a value and
will make libcurl return error for it.
1 was never a sensible value to use in production but was introduced
back in the days to help debugging. It was always documented clearly
this way.
1 was never supported by all SSL backends in libcurl, so this cleanup
makes the treatment of it unified.
The report's list of mistakes for this option were all PHP code and
while there's a binding layer between libcurl and PHP, the PHP team has
decided that they have an as thin layer as possible on top of libcurl so
they will not alter or specifically filter a 'TRUE' value for this
particular option. I sympathize with that position.
[1] = http://daniel.haxx.se/blog/2012/10/25/libcurl-claimed-to-be-dangerous/
Previously the curl_multi interface would freeze if darwinssl was
enabled and at least one of the handles tried to connect to a Web site
using HTTPS. Removed the "wouldblock" state darwinssl was using because
I figured out a solution for our "would block but in which direction?"
dilemma.
- Renamed st_ function prefix to darwinssl_
- Renamed Curl_st_ function prefix to Curl_darwinssl_
- Moved the duplicated ssl_connect_done out of the #ifdef in lib/urldata.h
- Fixed a teensy little bug that made non-blocking connection attempts block
- Made it so that it builds cleanly against the iOS 5.1 SDK
Building with CyaSSL failed compilation. Reason being that OCSP_REQUEST and
OCSP_RESPONSE are enum values in CyaSSL and defines in <wincrypt.h> included
via <winldap.h> in ldap.c.
http://curl.haxx.se/mail/lib-2012-06/0196.html
curl_sspi.c: Fixed mingw32-gcc compiler warnings
curl_sspi.c: Fixed length of error code hex output
The hex value was printed as signed 64-bit value on 64-bit systems:
SEC_E_WRONG_PRINCIPAL (0xFFFFFFFF80090322)
It is now correctly printed as the following:
SEC_E_WRONG_PRINCIPAL (0x80090322)
curl_sspi.c: Fallback to security function table version number
Instead of reporting an unknown version, the interface version is used.
curl_sspi.c: Removed SSPI/ version prefix from Curl_sspi_version
curl_schannel: Replaced static buffer sizes with defined names
curl_schannel.c: First brace when declaring functions on column 0
curl_schannel.c: Put the pointer sign directly at variable name
curl_schannel.c: Use structs directly instead of typedef'ed structs
curl_schannel.c: Removed space before opening brace
curl_schannel.c: Fixed lines being longer than 80 chars
The refactoring of HTTP CONNECT handling in commit 41b0237834 that
made it protocol independent broke it for the multi interface. This fix
now introduce a better state handling and moved some logic to the
http_proxy.c source file.
Reported by: Yang Tse
Bug: http://curl.haxx.se/mail/lib-2012-03/0162.html
Added a new CURLOPT_MAIL_AUTH option that allows the calling program to
set the optional AUTH parameter in the MAIL FROM command.
When this option is specified and an authentication mechanism is used
to communicate with the mail server then the AUTH parameter will be
included in the MAIL FROM command. This is particularly useful when the
calling program is acting as a relay in a trusted environment and
performing server to server communication, as it allows the relaying
server to specify the address of the mailbox that was used to
authenticate and send the original email.
Modify configure.ac to test for new CyaSSL Init function and remove
default install path to system. Change to CyaSSL OpenSSL header and
proper Init in code as well.
Note that this no longer detects or works with CyaSSL before v2
Allow an appliction to set libcurl specific SSL options. The first and
only options supported right now is CURLSSLOPT_ALLOW_BEAST.
It will make libcurl to disable any work-arounds the underlying SSL
library may have to address a known security flaw in the SSL3 and TLS1.0
protocol versions.
This is a reaction to us unconditionally removing that behavior after
this security advisory:
http://curl.haxx.se/docs/adv_20120124B.html
... it did however cause a lot of programs to fail because of old
servers not liking this work-around. Now programs can opt to decrease
the security in order to interoperate with old servers better.
This adds three new options to control the behavior of TCP keepalives:
- CURLOPT_TCP_KEEPALIVE: enable/disable probes
- CURLOPT_TCP_KEEPIDLE: idle time before sending first probe
- CURLOPT_TCP_KEEPINTVL: delay between successive probes
While not all operating systems support the TCP_KEEPIDLE and
TCP_KEEPINTVL knobs, the library will still allow these options to be
set by clients, silently ignoring the values.
"wait_data_conn" was added to the connectionbits in commit c834213ad5 for
handling active FTP connections but as it is purely FTP specific and now
only ever accessed by ftp.c I moved it into the FTP connection struct.
Backpedaled out the funny double-change of state in the multi state
machine by adding a new argument to the do_more() function to signal
completion. This way it can remain in the DO_MORE state properly until
done. Long term, the entire DO_MORE logic should be moved into the FTP
code and be hidden from the multi code as the logic is only used for
FTP.
1- Two new error codes are introduced.
CURLE_FTP_ACCEPT_FAILED to be set whenever ACCEPTing fails because of
FTP server connected.
CURLE_FTP_ACCEPT_TIMEOUT to be set whenever ACCEPTing timeouts.
Neither of these errors are considered fatal and control connection
remains OK because it could just be a firewall blocking server to
connect to the client.
2- One new setopt option was introduced.
CURLOPT_ACCEPTTIMEOUT_MS
It sets the maximum amount of time FTP client is going to wait for a
server to connect. Internal default accept timeout is 60 seconds.
Keep track of which sockets that are the result of accept() calls and
refuse to call the closesocket callback for those sockets. Test case 596
now verifies that the open socket callback is called the same number of
times as the closed socket callback for active FTP connections.
Bug: http://curl.haxx.se/mail/lib-2011-12/0018.html
Reported by: Gokhan Sengun
By setting PROTOPT_NOURLQUERY in the protocol handler struct, the
protocol will get the "query part" of the URL cut off before the data is
handled by the protocol-specific code. This makes libcurl adhere to
RFC3986 section 2.2.
Test 1220 is added to verify a file:// URL with query-part.
Now called 'use_ssl' instead, which better matches the current CURLOPT
name and since the option is used for all pingpong protocols (at least)
it makes sense to not use 'ftp' in the name.
After a PORT has been issued, and the multi handle would switch to the
CURLM_STATE_DO_MORE state (which is unique for FTP), libcurl would
return the wrong fdset to wait for when curl_multi_fdset() is
called. The code would blindly assume that it was waiting for a connect
of the second connection, while that isn't true immediately after the
PORT command.
Also, the function multi.c:domore_getsock() was highly FTP-centric and
therefore ugly to keep in protocol-agnostic code. I solved this problem
by introducing a new function pointer in the Curl_handler struct called
domore_getsock() which is only called during the DOMORE state for
protocols that set that pointer.
The new ftp.c:ftp_domore_getsock() function now returns fdset info about
the control connection's command/response handling while such a state is
in use, and goes over to waiting for a writable second connection first
once the commands are done.
The original problem could be seen by running test 525 and checking the
time stamps in the FTP server log. I can verify that this fix at least
fixes this problem.
Bug: http://curl.haxx.se/mail/lib-2011-10/0250.html
Reported by: Gokhan Sengun
Configure script option --enable-wb-ntlm-auth renamed to --enable-ntlm-wb
Configure script option --disable-wb-ntlm-auth renamed to --disable-ntlm-wb
Preprocessor symbol WINBIND_NTLM_AUTH_ENABLED renamed to NTLM_WB_ENABLED
Preprocessor symbol WINBIND_NTLM_AUTH_FILE renamed to NTLM_WB_FILE
Test harness env var CURL_NTLM_AUTH renamed to CURL_NTLM_WB_FILE
Static function wb_ntlm_close renamed to ntlm_wb_cleanup
Static function wb_ntlm_initiate renamed to ntlm_wb_init
Static function wb_ntlm_response renamed to ntlm_wb_response
When using the multi interface, a SOCKS proxy, and a connection that
wouldn't immediately consider itself connected (which my Linux tests do
by default), libcurl would be tricked into doing _two_ connects to the
SOCKS proxy when it setup the data connection and then of course the
second attempt would fail miserably and cause error.
This problem is a regression that was introduced by commit
4a42e5cdaa that was introduced in the 7.21.7 release.
Bug: http://curl.haxx.se/mail/lib-2011-08/0199.html
Reported by: Fabian Keil
Moved NTLMSSP_SIGNATURE, HOSTNAME_MAX, SHORTPAIR and LONGQUARTET definitions in ready for move to curl_ntlm.c.
Used separate variables for Windows SSPI and native code to ease moving of code to curl_ntlm.c.
Fixed typographical erros where SPPI should be SSPI.
Fixed compilation warnings on 64-bit builds when calling Windows SSPI functions.
Use preprocessor symbols WINBIND_NTLM_AUTH_ENABLED and WINBIND_NTLM_AUTH_FILE
for Samba's winbind daemon ntlm_auth helper code implementation and filename.
Retain preprocessor symbol USE_NTLM_SSO for NTLM single-sign-on feature
availability implementation independent.
For test harness, prefix NTLM_AUTH environment vars with CURL_
Refactor and rename configure option --with-ntlm-auth to --enable-wb-ntlm-auth[=FILE]
Introduced the initial setup to allow closesocket callbacks by making
sure sclose() is only ever called from one place in the libcurl source
and still run all test cases fine.
The protocol handler's flags field now can set that the protocol
requires a password, so that the set_userpass function doesn't have to
have the specific knowledge of which protocols that do.
Added CURLOPT_TRANSFER_ENCODING as the option to set to request Transfer
Encoding in HTTP requests (if built zlib enabled). I also renamed
CURLOPT_ENCODING to CURLOPT_ACCEPT_ENCODING (while keeping the old name
around) to reduce the confusion when we have to encoding options for
HTTP.
--tr-encoding is now the new command line option for curl to request
this, and thus I updated the test cases accordingly.
Since this struct member is used in the code to determine what and how
to decode automatically and since it is now also used for compressed
Transfer-Encodings, I renamed it to the more suitable 'auto_decoding'
Transfer-Encoding differs from Content-Encoding in a few subtle ways,
but primarily it concerns the transfer only and not the content so when
discovered to be compressed we know we have to uncompress it. There will
only arrive compressed transfers in a response after we have requested
them with the appropriate TE: header.
Test case 1122 and 1123 verify.
When asked to bind the local end of a connection when doing a request,
the code will now disqualify other existing connections from re-use even
if they are connected to the correct remote host.
This will also affect which connections that can be used for pipelining,
so that only connections that aren't bound or bound to the same
device/port you're asking for will be considered.
The PROT_* set of internal defines for the protocols is no longer
used. We now use the same bits internally as we have defined in the
public header using the CURLPROTO_ prefix. This is for simplicity and
because the PROT_* prefix was already used duplicated internally for a
set of KRB4 values.
The PROTOPT_* defines were moved up to just below the struct definition
within which they are used.
The protocol handler struct got a 'flags' field for special information
and characteristics of the given protocol.
This now enables us to move away central protocol information such as
CLOSEACTION and DUALCHANNEL from single defines in a central place, out
to each protocol's definition. It also made us stop abusing the protocol
field for other info than the protocol, and we could start cleaning up
other protocol-specific things by adding flags bits to set in the
handler struct.
The "protocol" field connectdata struct was removed as well and the code
now refers directly to the conn->handler->protocol field instead. To
make things work properly, the code now always store a conn->given
pointer that points out the original handler struct so that the code can
learn details from the original protocol even if conn->handler is
modified along the way - for example when switching to go over a HTTP
proxy.
Some protocols have to call the underlying functions without regard to
what exact state the socket signals. For example even if the socket says
"readable", the send function might need to be called while uploading,
or vice versa. This is the case for libssh2 based protocols: SCP and
SFTP and we now introduce a define to set those protocols and we make
the multi interface code aware of this concept.
This is another fix to make test 582 run properly.
Both SFTP and SCP are protocols that need to shut down stuff properly
when the connection is about to get torned down. The primary effect of
not doing this shows up as memory leaks (when using SCP or SFTP with the
multi interface).
This is one of the problems detected by test 582.
When using the multi interface and connecting to a host name that
resolves to multiple IP addresses, there was no logic that made it
continue to the next IP if connecting to the first address times
out. This is now corrected.
Make the c-ares resolver code ask for both IPv4 and IPv6 addresses when
IPv6 is enabled.
This is a workaround for the missing ares_getaddrinfo() and is a lot
easier to implement.
Note that as long as c-ares returns IPv4 addresses when IPv6 addresses
were requested but missing, this will cause a host's IPv4 addresses to
occur twice in the DNS cache.
URL: http://curl.haxx.se/mail/lib-2010-12/0041.html
The public axTLS header (at least as of 1.2.7) redefines the memory
functions. We #undef those again immediately after the public header to
limit the damage. This should be fixed in axTLS.
Added axTLS to autotool files and glue code to misc other files.
axtls.h maps SSL API functions, but may change.
axtls.c is just a stub file and will definitely change.
It helps to prevent a hangup with some FTP servers in case idle session
timeout has exceeded. But it may be useful also for other protocols
that send any quit message on disconnect. Currently used by FTP, POP3,
IMAP and SMTP.
While changing Curl_sec_read_msg to accept an enum protection_level
instead of an int, I went ahead and fixed the usage of the associated
fields.
Some code was assuming that prot_clear == 0. Fixed those to use the
proper value. Added assertions prior to any code that would set the
protection level.
The IP version choice was previously only in the UserDefined struct
within the SessionHandle, but since we sometimes alter that option
during a request we need to have it on a per-connection basis.
I also moved more "init conn" code into the allocate_conn() function
which is designed for that purpose more or less.
CURLOPT_RESOLVE is a new option that sends along a curl_slist with
name:port:address sets that will populate the DNS cache with entries so
that request can be "fooled" to use another host than what otherwise
would've been used. Previously we've encouraged the use of Host: for
that when dealing with HTTP, but this new feature has the added bonus
that it allows the name from the URL to be used for TLS SNI and server
certificate name checks as well.
This is a first change. Surely more will follow to make it decent.
When given a custom host name in a Host: header, we can use it for
several different purposes other than just cookies, so we rename it and
use it for SSL SNI etc.
HTTP allows that a server sends trailing headers after all the chunks
have been sent WITHOUT signalling their presence in the first response
headers. The "Trailer:" header is only a SHOULD there and as we need to
handle the situation even without that header I made libcurl ignore
Trailer: completely.
Test case 1116 was added to verify this and to make sure we handle more
than one trailer header properly.
Reported by: Patrick McManus
Bug: http://curl.haxx.se/bug/view.cgi?id=3052450
Curl_expire() is now expanded to hold a list of timeouts for each easy
handle. Only the closest in time will be the one used as the primary
timeout for the handle and will be used for the splay tree (which sorts
and lists all handles within the multi handle).
When the main timeout has triggered/expired, the next timeout in time
that is kept in the list will be moved to the main timeout position and
used as the key to splay with. This way, all timeouts that are set with
Curl_expire() internally will end up as a proper timeout. Previously any
Curl_expire() that set a _later_ timeout than what was already set was
just silently ignored and thus missed.
Setting Curl_expire() with timeout 0 (zero) will cancel all previously
added timeouts.
Corrects known bug #62.
Test 563 is enabled now and verifies that the combo FTP type=A URL,
CURLOPT_PORT set and proxy work fine. As a bonus I managed to remove the
somewhat odd FTP check in parse_remote_port() and instead converted it
to a better and more generic 'slash_removed' struct field. Checking the
->protocol field isn't right since when an FTP:// URL is sent over a
HTTP proxy, the protocol is HTTP but the URL was handled by the FTP code
and thus slash_removed is set TRUE for this case.
"The BSD version of PolarSSL was made for migratory purposes only and is not
maintained. The GPL version of PolarSSL is actually the only actively
developed version, so I would be very reluctant to use the BSD version." /
Paul Bakker, PolarSSL hacker.
Signed-off-by: Hoi-Ho Chan <hoiho.chan@gmail.com>
FTP(S) use two connections that can be set to different recv and
send functions independently, so by introducing recv+send pairs
in the same manner we already have sockets/connections we can
work with FTPS fine.
This commit fixes the FTPS regression introduced in change d64bd82.
Howard Chu brought the bulk work of this patch that properly
moves out the sending and recving of data to the parts of the
code that are properly responsible for the various ways of doing
so.
Daniel Stenberg assisted with polishing a few bits and fixed some
minor flaws in the original patch.
Another upside of this patch is that we now abuse CURLcodes less
with the "magic" -1 return codes and instead use CURLE_AGAIN more
consistently.
This is Hoi-Ho Chan's patch with some minor fixes by me. There
are some potential issues in this, but none worse than we can
sort out on the list and over time.
The main change is to allow input from user-specified methods,
when they are specified with CURLOPT_READFUNCTION.
All calls to fflush(stdout) in telnet.c were removed, which makes
using 'curl telnet://foo.com' painful since prompts and other data
are not always returned to the user promptly. Use
'curl --no-buffer telnet://foo.com' instead. In general,
the user should have their CURLOPT_WRITEFUNCTION do a fflush
for interactive use.
Also fix assumption that reading from stdin never returns < 0.
Old code could crash in that case.
Call progress functions in telnet main loop.
Signed-off-by: Ben Greear <greearb@candelatech.com>
from hostip.h to setup.h in order to allow proper inclusion in any file.
This represents no functional change at all in which resolver is used,
everything still works as usual, internally and externally there is no
difference in behavior.
command is a special "hack" used by the drftpd server, but even though it is
a custom extension I've deemed it fine to add to libcurl since this server
seems to survive and people keep using it and want libcurl to support
it. The new libcurl option is named CURLOPT_FTP_USE_PRET, and it is also
usable from the curl tool with --ftp-pret. Using this option on a server
that doesn't support this command will make libcurl fail.
rework patch that now integrates TFTP properly into libcurl so that it can
be used non-blocking with the multi interface and more. BLKSIZE also works.
The --tftp-blksize option was added to allow setting the TFTP BLKSIZE from
the command line.
transfer.c for blocking. It is currently used only by SCP and SFTP protocols.
This enhancement resolves an issue with 100% CPU usage during SFTP upload,
reported by Vourhey.
Fix SIGSEGV on free'd easy_conn when pipe unexpectedly breaks
Fix data corruption issue with re-connected transfers
Fix use after free if we're completed but easy_conn not NULL
in NSS-powered libcurl. Now the client certificates can be selected
automatically by a NSS built-in hook. Additionally pre-login to all PKCS11
slots is no more performed. It used to cause problems with HW tokens.
- Fixed reference counting for NSS client certificates. Now the PEM reader
module should be always properly unloaded on Curl_nss_cleanup(). If the unload
fails though, libcurl will try to reuse the already loaded instance.
KEEP_RECV to better match the general terminology: receive and send is what we
do from the (remote) servers. We read and write from and to the local fs.
out to be leaking cacerts. Kamil Dudka helped me complete the fix. The issue
is found in Redhat's bug tracker:
https://bugzilla.redhat.com/show_bug.cgi?id=453612
There are still memory leaks present, but they seem to have other reasons.
(http://curl.haxx.se/docs/adv_20090303.html also known as CVE-2009-0037) in
which previous libcurl versions (by design) can be tricked to access an
arbitrary local/different file instead of a remote one when
CURLOPT_FOLLOWLOCATION is enabled. This flaw is now fixed in this release
together this the addition of two new setopt options for controlling this
new behavior:
o CURLOPT_REDIR_PROTOCOLS controls what protocols libcurl is allowed to
follow to when CURLOPT_FOLLOWLOCATION is enabled. By default, this option
excludes the FILE and SCP protocols and thus you nee to explicitly allow
them in your app if you really want that behavior.
o CURLOPT_PROTOCOLS controls what protocol(s) libcurl is allowed to fetch
using the primary URL option. This is useful if you want to allow a user or
other outsiders control what URL to pass to libcurl and yet not allow all
protocols libcurl may have been built to support.
FTP with the multi interface: when a transfer fails, like when aborted by a
write callback, the control connection was wrongly closed and thus not
re-used properly.
This change is also an attempt to cleanup the code somewhat in this area, as
now the FTP code attempts to keep (better) track on pending responses
necessary to get read in ftp_done().
plain FTP connections, and it will then allow MKD to fail once and retry the
CWD afterwards. This is especially useful if you're doing many simultanoes
connections against the same server and they all have this option enabled,
as then CWD may first fail but then another connection does MKD before this
connection and thus MKD fails but trying CWD works! The numbers can
(should?) now be set with the convenience enums now called
CURLFTP_CREATE_DIR and CURLFTP_CREATE_DIR_RETRY.
Tests has proven that if you're making an application that uploads a set of
files to an ftp server, you will get a noticable gain in speed if you're
using multiple connections and this option will be then be very useful.
the condition in the previous request was unmet. This is typically a time
condition set with CURLOPT_TIMECONDITION and was previously not possible to
reliably figure out. From bug report #2565128
(http://curl.haxx.se/bug/view.cgi?id=2565128)
Curl_sspi_global_init() and Curl_sspi_global_cleanup() which previously were
named Curl_ntlm_global_init() and Curl_ntlm_global_cleanup() in http_ntlm.c
Also adjusted socks_sspi.c to remove the link-time dependency on the Windows
SSPI library using it now in the same way as it was done in http_ntlm.c.
CURLOPT_SOCKS5_GSSAPI_SERVICE and CURLOPT_SOCKS5_GSSAPI_NEC to allow libcurl
to do GSS-style authentication with SOCKS5 proxies. The curl tool got the
options called --socks5-gssapi-service and --socks5-gssapi-nec to enable
these.
They basically offer the same thing the NO_PROXY environment variable only
offered previously: list a set of host names that shall not use the proxy
even if one is specified.
now has an improved ability to do right when the multi interface (both
"regular" and multi_socket) is used for SCP and SFTP transfers. This should
result in (much) less busy-loop situations and thus less CPU usage with no
speed loss.
there are servers "out there" that relies on the client doing this broken
Digest authentication. Apache even comes with an option to work with such
broken clients.
The difference is only for URLs that contain a query-part (a '?'-letter and
text to the right of it).
libcurl now supports this quirk, and you enable it by setting the
CURLAUTH_DIGEST_IE bit in the bitmask you pass to the CURLOPT_HTTPAUTH or
CURLOPT_PROXYAUTH options. They are thus individually controlled to server
and proxy.
(http://curl.haxx.se/bug/view.cgi?id=2221237) that identified an infinite
loop during GSS authentication given some specific conditions. With his
patience and great feedback I managed to narrow down the problem and
eventually fix it although I can't test any of this myself!
problem with my CURLINFO_PRIMARY_IP fix from October 7th that caused a NULL
pointer read. I also took the opportunity to clean up this logic (storing of
the connection's IP address) somewhat as we had it stored in two different
places and ways previously and they are now unified.
make CURLOPT_PROXYUSERPWD sort of deprecated. The primary motive for adding
these new options is that they have no problems with the colon separator
that the CURLOPT_PROXYUSERPWD option does.
curl_easy_setopt: CURLOPT_USERNAME and CURLOPT_PASSWORD that sort of
deprecates the good old CURLOPT_USERPWD since they allow applications to set
the user name and password independently and perhaps more importantly allow
both to contain colon(s) which CURLOPT_USERPWD doesn't fully support.
CURLOPT_POST301 (but adds a define for backwards compatibility for you who
don't define CURL_NO_OLDIES). This option allows you to now also change the
libcurl behavior for a HTTP response 302 after a POST to not use GET in the
subsequent request (when CURLOPT_FOLLOWLOCATION is enabled). I edited the
patch somewhat before commit. The curl tool got a matching --post302
option. Test case 1076 was added to verify this.
enabling this feature with CURLOPT_CERTINFO for a request using SSL (HTTPS
or FTPS), libcurl will gather lots of server certificate info and that info
can then get extracted by a client after the request has completed with
curl_easy_getinfo()'s CURLINFO_CERTINFO option. Linus Nielsen Feltzing
helped me test and smoothen out this feature.
Unfortunately, this feature currently only works with libcurl built to use
OpenSSL.
This feature was sponsored by networking4all.com - thanks!
parser to allow numerical IPv6-addresses to be specified with the scope
given, as per RFC4007 - with a percent letter that itself needs to be URL
escaped. For example, for an address of fe80::1234%1 the HTTP URL is:
"http://[fe80::1234%251]/"
CURLINFO_APPCONNECT_TIME. This is set with the "application layer"
handshake/connection is completed (typically SSL, TLS or SSH). By using this
you can figure out the application layer's own connect time. You can extract
the time stamp using curl's -w option and the new variable named
'time_appconnect'. This feature was sponsored by Lenny Rachitsky at NeuStar.
curl_easy_getinfo. It returns a pointer to a string with the most recently
used IP address. Modified test case 500 to also verify this feature. The
implementing of this feature was sponsored by Lenny Rachitsky at NeuStar.
redirections and thus cannot use CURLOPT_FOLLOWLOCATION easily, we now
introduce the new CURLINFO_REDIRECT_URL option that lets applications
extract the URL libcurl would've redirected to if it had been told to. This
then enables the application to continue to that URL as it thinks is
suitable, without having to re-implement the magic of creating the new URL
from the Location: header etc. Test 1029 verifies it.
