Introducing a number of options to the multi interface that
allows for multiple pipelines to the same host, in order to
optimize the balance between the penalty for opening new
connections and the potential pipelining latency.
Two new options for limiting the number of connections:
CURLMOPT_MAX_HOST_CONNECTIONS - Limits the number of running connections
to the same host. When adding a handle that exceeds this limit,
that handle will be put in a pending state until another handle is
finished, so we can reuse the connection.
CURLMOPT_MAX_TOTAL_CONNECTIONS - Limits the number of connections in total.
When adding a handle that exceeds this limit,
that handle will be put in a pending state until another handle is
finished. The free connection will then be reused, if possible, or
closed if the pending handle can't reuse it.
Several new options for pipelining:
CURLMOPT_MAX_PIPELINE_LENGTH - Limits the pipeling length. If a
pipeline is "full" when a connection is to be reused, a new connection
will be opened if the CURLMOPT_MAX_xxx_CONNECTIONS limits allow it.
If not, the handle will be put in a pending state until a connection is
ready (either free or a pipe got shorter).
CURLMOPT_CONTENT_LENGTH_PENALTY_SIZE - A pipelined connection will not
be reused if it is currently processing a transfer with a content
length that is larger than this.
CURLMOPT_CHUNK_LENGTH_PENALTY_SIZE - A pipelined connection will not
be reused if it is currently processing a chunk larger than this.
CURLMOPT_PIPELINING_SITE_BL - A blacklist of hosts that don't allow
pipelining.
CURLMOPT_PIPELINING_SERVER_BL - A blacklist of server types that don't allow
pipelining.
See the curl_multi_setopt() man page for details.
Updated the POP3 sub-section to refer to message ID rather than mailbox.
Added an IMAP sub-section with example URLs depicting the specification
of mailbox, uid and section.
* Elaborates on default values of some curl_easy_setopt() options.
* Reminds the user to cast variadic arguments to curl_easy_setopt() to
'void *' where curl internally interprets them as such.
* Clarifies the working of the CURLOPT_SEEKFUNCTION option for
curl_easy_setopt().
* Fixes typo 'forth' → 'fourth'.
* Elaborates on CURL_SOCKET_TIMEOUT.
* Adds some missing periods.
* Notes that the return value of curl_version() must not be passed to
free().
* Adds several links to documentation of library functions which were
missing.
* Marks documentation of deprecated library functions "(deprecated)".
* Removes spurious .html suffixes.
Schannel and darwinssl use the certificates built into the
OS to do vert verification instead of bundles. darwinssl
is thread-safe. Corrected typos in the NSS docs.
After a research team wrote a document[1] that found several live source
codes out there in the wild that misused the CURLOPT_SSL_VERIFYHOST
option thinking it was a boolean, this change now bans 1 as a value and
will make libcurl return error for it.
1 was never a sensible value to use in production but was introduced
back in the days to help debugging. It was always documented clearly
this way.
1 was never supported by all SSL backends in libcurl, so this cleanup
makes the treatment of it unified.
The report's list of mistakes for this option were all PHP code and
while there's a binding layer between libcurl and PHP, the PHP team has
decided that they have an as thin layer as possible on top of libcurl so
they will not alter or specifically filter a 'TRUE' value for this
particular option. I sympathize with that position.
[1] = http://daniel.haxx.se/blog/2012/10/25/libcurl-claimed-to-be-dangerous/
This is a minor change in behavior after having been pointed out by Mark
Tully and discussed on the list. Initially this case would internally
call poll() with no sockets and a timeout which would equal a sleep for
that specified time.
Bug: http://curl.haxx.se/mail/lib-2012-10/0076.html
Reported by: Mark Tully
"Currently you can only share DNS and/or COOKIE data" is incorrect since
also SSL sessions can be shared.
Bug: http://curl.haxx.se/bug/view.cgi?id=3562261
Reported by: Joe Mason
- For all *FUNCTION options, they now all show the complete prototype in
the description. Previously some of them would just refer to a
typedef'ed function pointer in the curl.h header.
- I made the phrasing of that "Pass a pointer to a function that matches
the following prototype" the same for all *FUNCTION option descriptions.
- I removed some uses of 'should'. I think I sometimes over-use this
word as in many places I actually mean MUST or otherwise more specific
and not-so-optional synonyms.
Setting bit 2 for this value was documented as having a constant value
defined as CURL_REDIR_POST_303 yet referenced a 302 request.
Additionally corrected the meaning of CURL_REDIR_POST_ALL for all three
bits and fixed problems with the bolding of keywords in this section.
Standardised how RFCs are referenced so that the website may autolink to
the correct documentation on ietf.org. Additionally removed the one link
to RFC3986 on curl.haxx.se.
If an empty string is passed to CURLOPT_SSH_PUBLIC_KEYFILE, libcurl will
pass no public key to libssh2 which then tries to compute it from the
private key. This is known to work when libssh2 1.4.0+ is linked against
OpenSSL.
Set the conn->data->info.httpcode variable in smtp_statemach_act() to
allow Curl_getinfo() to return the SMTP response code via the
CURLINFO_RESPONSE_CODE action.
Added information relating to the new CURLOPT_MAIL_AUTH parameter and
reworked CURLOPT_MAIL_FROM and CURLOPT_MAIL_RCPT to be a clearer.
Fixed inconsistencies of "vocalisation of the abbreviation" versus
"vocalisation of the first word" for all abbreviations.
Corrected a typo in CURLOPT_NOPROXY.
Allow an appliction to set libcurl specific SSL options. The first and
only options supported right now is CURLSSLOPT_ALLOW_BEAST.
It will make libcurl to disable any work-arounds the underlying SSL
library may have to address a known security flaw in the SSL3 and TLS1.0
protocol versions.
This is a reaction to us unconditionally removing that behavior after
this security advisory:
http://curl.haxx.se/docs/adv_20120124B.html
... it did however cause a lot of programs to fail because of old
servers not liking this work-around. Now programs can opt to decrease
the security in order to interoperate with old servers better.
This adds three new options to control the behavior of TCP keepalives:
- CURLOPT_TCP_KEEPALIVE: enable/disable probes
- CURLOPT_TCP_KEEPIDLE: idle time before sending first probe
- CURLOPT_TCP_KEEPINTVL: delay between successive probes
While not all operating systems support the TCP_KEEPIDLE and
TCP_KEEPINTVL knobs, the library will still allow these options to be
set by clients, silently ignoring the values.
1- Two new error codes are introduced.
CURLE_FTP_ACCEPT_FAILED to be set whenever ACCEPTing fails because of
FTP server connected.
CURLE_FTP_ACCEPT_TIMEOUT to be set whenever ACCEPTing timeouts.
Neither of these errors are considered fatal and control connection
remains OK because it could just be a firewall blocking server to
connect to the client.
2- One new setopt option was introduced.
CURLOPT_ACCEPTTIMEOUT_MS
It sets the maximum amount of time FTP client is going to wait for a
server to connect. Internal default accept timeout is 60 seconds.
Parameters were underquoted, resulting in
warning: AC_LANG_CONFTEST: no AC_LANG_SOURCE call detected in body
Signed-off-by: Bernhard Reutner-Fischer <rep.dot.nop@gmail.com>
Do not try to resolve interfaces names via DNS by recognizing interface
names in a few ways. If the interface option argument has a prefix of
"if!" then treat the argument as only an interface. Similarly, if the
interface argument is the name of an interface (even if it does not have
an IP address assigned), treat it as an interface name. Finally, if the
interface argument is prefixed by "host!" treat it as a hostname that
must be resolved by /etc/hosts or DNS.
These changes allow a client using the multi interfaces to avoid
blocking on name resolution if the interface loses its IP address or
disappears.
If the option is set to 0, the default timeout will be used - which in
modern libcurl versions equals 300 seconds (== 5 minutes).
Bug: http://curl.haxx.se/mail/lib-2011-12/0051.html
Reported by: Vladimir Grishchenko
When the new socket is created for an active connection, it is now done
using the open socket callback.
Test case 596 was modified to run fine, although it hides the fact that
the close callback is still called too many times, as it also gets
called for closing sockets that were created with accept().
Experience has shown that the symbols-in-versions file is very useful to
applications that want to build with a wide range of libcurl versions.
It is however easy to get it wrong and the source gets a bit messy with
all the fixed numerical comparisions.
The point of this script is to provide an easy-to-use macro for libcurl-
using applications to do preprocessor checks for specific libcurl
defines, and yet make the code clearly show what the macro is used for.
Added pop3 username and password example as well as an explanation of
how path part of the URL is used under pop3.
Additionally have corrected a couple of typos.
Don't even declare the struct members for disabled features
Introducing the CURLSHE_NOT_BUILT_IN return code for the share interface
when trying to set a sharing option that has been disabled (or not
enabled) in the library.
Slight rewording of the CURLOPT_URL SMTP sub-section.
Corrected the incorrect use of hyphens on the three uses of
"zero-terminated" with "zero terminated" to match the rest of the
document.
Corrected the use of an out of place hyphen in CURLOPT_NOPROXY section.
Allow (*curl_write_callback) write callbacks to return
CURL_WRITEFUNC_OUT_OF_MEMORY to properly indicate libcurl of OOM conditions
inside the callback itself.
If a socket is larger than FD_SETSIZE, avoid using FD_SET() on the
platforms where this is possible.
Bug: http://curl.haxx.se/bug/view.cgi?id=3413274
Reported by: Tim Starling
Using 'socks5h' as proxy protocol will make it a
CURLPROXY_SOCKS5_HOSTNAME proxy which is SOCKS5 and asking the proxy to
resolve host names. I found no "standard" protocol name for this.
Clarify that the '-', '.', '_' or '~' letters are also not escaped since
they shouldn't according to RFC3986 section 2.3.
This is how this function has behaved since sep 2010, commit
5df13c3173.
Added CURLOPT_TRANSFER_ENCODING as the option to set to request Transfer
Encoding in HTTP requests (if built zlib enabled). I also renamed
CURLOPT_ENCODING to CURLOPT_ACCEPT_ENCODING (while keeping the old name
around) to reduce the confusion when we have to encoding options for
HTTP.
--tr-encoding is now the new command line option for curl to request
this, and thus I updated the test cases accordingly.
Stop the abuse of CURLE_FAILED_INIT as return code for things not being
init related by introducing two new return codes:
CURLE_NOT_BUILT_IN and CURLE_UNKNOWN_OPTION
CURLE_NOT_BUILT_IN replaces return code 4 that has been obsoleted for
several years. It is used for returning error when something is
attempted to be used but the feature/option was not enabled or
explictitly disabled at build-time. Getting this error mostly means that
libcurl needs to be rebuilt.
CURLE_FAILED_INIT is now saved and used strictly for init
failures. Getting this problem means something went seriously wrong,
like a resource shortage or similar.
CURLE_UNKNOWN_OPTION is the option formerly known as
CURLE_UNKNOWN_TELNET_OPTION (and the old name is still present,
separately defined to be removed in a very distant future). This error
code is meant to be used to return when an option is given to libcurl
that isn't known. This problem would mostly indicate a problem in the
program that uses libcurl.
The read callback must return the exact requested amount of data when it
is used for doing TFTP uploads. This is due to how it deals with data
internally. This could/should be fixed but for now we document the
existing behavior.
Reported by: Colin Blair
Bug: http://curl.haxx.se/mail/lib-2011-03/0319.html
When NSS-powered libcurl connected to a SSL server with
CURLOPT_SSL_VERIFYPEER equal to zero, NSS remembered that the peer
certificate was accepted by libcurl and did not ask the second time when
connecting to the same server with CURLOPT_SSL_VERIFYPEER equal to one.
This patch turns off the SSL session cache for the particular SSL socket
if peer verification is disabled. In order to avoid any performance
impact, the peer verification is completely skipped in that case, which
makes it even faster than before.
Bug: https://bugzilla.redhat.com/678580
On second thought, I think CURLE_TLSAUTH_FAILED should be eliminated. It
was only being raised when an internal error occurred while allocating
or setting the GnuTLS SRP client credentials struct. For TLS
authentication failures, the general CURLE_SSL_CONNECT_ERROR seems
appropriate; its error string already includes "passwords" as a possible
cause. Having a separate TLS auth error code might also cause people to
think that a TLS auth failure means the wrong username or password was
entered, when it could also be a sign of a man-in-the-middle attack.
... and update the curl.1 and curl_easy_setopt.3 man pages such that
they do not suggest to use an OpenSSL utility if curl is not built
against OpenSSL.
Bug: https://bugzilla.redhat.com/669702
This is a meta symbol. OR this value together with a single specific
auth value to force libcurl to probe for un-restricted auth and if not,
only that single auth algorithm is acceptable.
For example you can use CURLAUTH_DIGEST|CURLAUTH_ONLY to make libcurl
first probe for what method to use, but yet only consider Digest to be
acceptable.
Using _only_ CURLAUTH_DIGEST without the CURLAUTH_ONLY field, will make
libcurl explicitly use Digest right away and not do any probing.
I've developed a script I call symbol-scan.pl that scans the curl.h and
multi.h header files and compare the symbols it finds in there with the
symbols symbols-in-versions documents and outputs a report on the
differences. Using this I've dug through the history to fill up
symbols-in-versions with all the symbols my script found mismatches for.
I will commit symbol-scan.pl separatly and think of a way to put it to
use in the build/tests so that we from now on will get this in-sync
check automatically.
The invocation of autoconf's AC_PATH_PROG( ) is not quite right for
finding curl-config. This fix corrects the negative case (where
curl-config is not found).
The macro provides a --with-libcurl option that expects a PREFIX to be
specified and not actually a "directory" in which libcurl will be found.
This now spells that out more clearly.
Reported by: Dan Locks
Bug: http://curl.haxx.se/bug/view.cgi?id=3079891
The numerical value passed to CURLOPT_RESUME_FROM for FTP uploads is
interpreted and used as position where to resume the _reading_ of the
local file and it will "blindly" append that data on the remote
file. This was certainly not clear in the docs previously.
Reported by: catalin
Bug: http://curl.haxx.se/bug/view.cgi?id=3048174
In some places where the name 'stream' has been used for naming a
function argument that is in fact settable with a setopt() option we now
call that argument 'userdata' to make it more obvious that it is in fact
possible to set by the application.
Suggested by: Jeff Pohlmeyer
The SOCKET type in Win64 is 64 bits large (and thus so is curl_socket_t
on that platform), and long is only 32 bits. It makes it impossible for
curl_easy_getinfo() to return a socket properly with the
CURLINFO_LASTSOCKET option as for all other operating systems.
curl_multi_timeout(3) is simply the wrong function to use
if you're using the multi_socket API and this document now
states this pretty clearly to help guiding users.
properly in angle brackets. Recipients provided with CURLOPT_MAIL_RCPT now
get angle bracket wrapping automatically by libcurl unless the recipient
starts with an angle bracket as then the app is assumed to deal with that
properly on its own.
simply check for CURLM_CALL_MULTI_PERFORM internally. This has the added
benefit that this goes in line with my long-term wishes to get rid of the
CURLM_CALL_MULTI_PERFORM all together from the public API.
command is a special "hack" used by the drftpd server, but even though it is
a custom extension I've deemed it fine to add to libcurl since this server
seems to survive and people keep using it and want libcurl to support
it. The new libcurl option is named CURLOPT_FTP_USE_PRET, and it is also
usable from the curl tool with --ftp-pret. Using this option on a server
that doesn't support this command will make libcurl fail.
With the curl memory tracking feature decoupled from the debug build feature,
CURLDEBUG and DEBUGBUILD preprocessor symbol definitions are used as follows:
CURLDEBUG used for curl debug memory tracking specific code (--enable-curldebug)
DEBUGBUILD used for debug enabled specific code (--enable-debug)
Chen pointed out how curl couldn't upload with resume when reading from a
pipe.
This ended up with the introduction of a new return code for the
CURLOPT_SEEKFUNCTION callback that basically says that the seek failed but
that libcurl may try to resolve the situation anyway. In our case this means
libcurl will attempt to instead read that much data from the stream instead
of seeking and that way curl can now upload with resume when data is read
from a stream!
and 1 on fatal errors. Previously it only mentioned non-zero on fatal
errors. This is a slight change in meaning, but it follows what we've done
elsewhere before and it opens up for LOTS of more useful return codes
whenever we can think of them...
(http://curl.haxx.se/docs/adv_20090303.html also known as CVE-2009-0037) in
which previous libcurl versions (by design) can be tricked to access an
arbitrary local/different file instead of a remote one when
CURLOPT_FOLLOWLOCATION is enabled. This flaw is now fixed in this release
together this the addition of two new setopt options for controlling this
new behavior:
o CURLOPT_REDIR_PROTOCOLS controls what protocols libcurl is allowed to
follow to when CURLOPT_FOLLOWLOCATION is enabled. By default, this option
excludes the FILE and SCP protocols and thus you nee to explicitly allow
them in your app if you really want that behavior.
o CURLOPT_PROTOCOLS controls what protocol(s) libcurl is allowed to fetch
using the primary URL option. This is useful if you want to allow a user or
other outsiders control what URL to pass to libcurl and yet not allow all
protocols libcurl may have been built to support.
CURLINFO_CONTENT_LENGTH_DOWNLOAD and CURLINFO_CONTENT_LENGTH_UPLOAD return
-1 if the sizes aren't know. Previously these returned 0, make it impossible
to detect the difference between actually zero and unknown.
plain FTP connections, and it will then allow MKD to fail once and retry the
CWD afterwards. This is especially useful if you're doing many simultanoes
connections against the same server and they all have this option enabled,
as then CWD may first fail but then another connection does MKD before this
connection and thus MKD fails but trying CWD works! The numbers can
(should?) now be set with the convenience enums now called
CURLFTP_CREATE_DIR and CURLFTP_CREATE_DIR_RETRY.
Tests has proven that if you're making an application that uploads a set of
files to an ftp server, you will get a noticable gain in speed if you're
using multiple connections and this option will be then be very useful.
the condition in the previous request was unmet. This is typically a time
condition set with CURLOPT_TIMECONDITION and was previously not possible to
reliably figure out. From bug report #2565128
(http://curl.haxx.se/bug/view.cgi?id=2565128)
version 1.1 instead of 1.0 like before. This change also introduces the new
proxy type for libcurl called 'CURLPROXY_HTTP_1_0' that then allows apps to
switch (back) to CONNECT 1.0 requests. The curl tool also got a --proxy1.0
option that works exactly like --proxy but sets CURLPROXY_HTTP_1_0.
I updated all test cases cases that use CONNECT and I tried to do some using
--proxy1.0 and some updated to do CONNECT 1.1 to get both versions run.
CURLOPT_SOCKS5_GSSAPI_SERVICE and CURLOPT_SOCKS5_GSSAPI_NEC to allow libcurl
to do GSS-style authentication with SOCKS5 proxies. The curl tool got the
options called --socks5-gssapi-service and --socks5-gssapi-nec to enable
these.
They basically offer the same thing the NO_PROXY environment variable only
offered previously: list a set of host names that shall not use the proxy
even if one is specified.
there are servers "out there" that relies on the client doing this broken
Digest authentication. Apache even comes with an option to work with such
broken clients.
The difference is only for URLs that contain a query-part (a '?'-letter and
text to the right of it).
libcurl now supports this quirk, and you enable it by setting the
CURLAUTH_DIGEST_IE bit in the bitmask you pass to the CURLOPT_HTTPAUTH or
CURLOPT_PROXYAUTH options. They are thus individually controlled to server
and proxy.
make CURLOPT_PROXYUSERPWD sort of deprecated. The primary motive for adding
these new options is that they have no problems with the colon separator
that the CURLOPT_PROXYUSERPWD option does.
curl_easy_setopt: CURLOPT_USERNAME and CURLOPT_PASSWORD that sort of
deprecates the good old CURLOPT_USERPWD since they allow applications to set
the user name and password independently and perhaps more importantly allow
both to contain colon(s) which CURLOPT_USERPWD doesn't fully support.
enabling this feature with CURLOPT_CERTINFO for a request using SSL (HTTPS
or FTPS), libcurl will gather lots of server certificate info and that info
can then get extracted by a client after the request has completed with
curl_easy_getinfo()'s CURLINFO_CERTINFO option. Linus Nielsen Feltzing
helped me test and smoothen out this feature.
Unfortunately, this feature currently only works with libcurl built to use
OpenSSL.
This feature was sponsored by networking4all.com - thanks!
parser to allow numerical IPv6-addresses to be specified with the scope
given, as per RFC4007 - with a percent letter that itself needs to be URL
escaped. For example, for an address of fe80::1234%1 the HTTP URL is:
"http://[fe80::1234%251]/"
non-zero with the fixed value of 1. We should strive at making options
support '1' for enabling them mentioned explicitly, as that then will allow
us for to extend them in the future without breaking older programs.
curl-library list on July 9th 2008 by Mathew Hounsell)
NOTE: the name resolve functions of various libc implementations don't re-read
name server information unless explicitly told so (by for example calling
Ires_init(3). This may cause libcurl to keep using the older server even
if DHCP has updated the server info, and this may look like a DNS cache issue
to the casual libcurl-app user.
CURLINFO_APPCONNECT_TIME. This is set with the "application layer"
handshake/connection is completed (typically SSL, TLS or SSH). By using this
you can figure out the application layer's own connect time. You can extract
the time stamp using curl's -w option and the new variable named
'time_appconnect'. This feature was sponsored by Lenny Rachitsky at NeuStar.
