This uses the brotli external library (https://github.com/google/brotli).
Brotli becomes a feature: additional curl_version_info() bit and
structure fields are provided for it and CURLVERSION_NOW bumped.
Tests 314 and 315 check Brotli content unencoding with correct and
erroneous data.
Some tests are updated to accomodate with the now configuration dependent
parameters of the Accept-Encoding header.
This is implemented as an output streaming stack of unencoders, the last
calling the client write procedure.
New test 230 checks this feature.
Bug: https://github.com/curl/curl/pull/2002
Reported-By: Daniel Bankhead
By properly keeping track of the last entry in the list of URLs/uploads
to handle, curl now avoids many meaningless traverses of the list which
speeds up many-URL handling *MASSIVELY* (several magnitudes on 100K
URLs).
Added test 1291, to verify that it doesn't take ages - but we don't have
any detection of "too slow" command in the test suite.
Reported-by: arainchik on github
Fixes#1959Closes#2052
Also upgrade test 1133 to cover this case and clarify man page about
form data quoting.
Bug: https://github.com/curl/curl/issues/2022
Reported-By: omau on github
If stdin is not a regular file, its content is memory-buffered to enable
a possible data "rewind".
In all cases, stdin data size is determined before real use to avoid
having an unknown part's size.
--libcurl generated code is left as an unbuffered stdin fread/fseek callback
part with unknown data size.
Buffering is not supported in deprecated curl_formadd() API.
This reverts commit f3e03f6c0a.
Caused memory leaks in the fuzzer, needs to be done differently.
Disable test 1553 for now too, as it causes memory leaks without this
commit!
When imap_done() got called before a connection is setup, it would try
to "finish up" and dereffed a NULL pointer.
Test case 1553 managed to reproduce. I had to actually use a host name
to try to resolve to slow it down, as using the normal local server IP
will make libcurl get a connection in the first curl_multi_perform()
loop and then the bug doesn't trigger.
Fixes#1953
Assisted-by: Max Dymond
... fixes a memory leak with at least IMAP when remove_handle is never
called and the transfer is abruptly just abandoned early.
Test 1552 added to verify
Detected by OSS-fuzz
Assisted-by: Max Dymond
Closes#1954
... a single double quote could leave the entry path buffer without a zero
terminating byte. CVE-2017-1000254
Test 1152 added to verify.
Reported-by: Max Dymond
Bug: https://curl.haxx.se/docs/adv_20171004.html
When curl and libcurl are built with some protocols disabled, they stop
setting and receiving some options that don't make sense with those
protocols. In particular, when HTTP is disabled many options aren't set
that are used only by HTTP. However, some options that appear to be
HTTP-only are actually used by other protocols as well (some despite
having HTTP in the name) and should be set, but weren't. This change now
causes some of these options to be set and used for more (or for all)
protocols. In particular, this fixes tests 646 through 649 in an
HTTP-disabled build, which use the MIME API in the mail protocols.
... as the test cases themselves do that and it makes it easier to add
crazy test cases.
Test 800 updated to use user name + password that need quoting.
Test 856 updated to trigger an auth fail differently.
Ref: #1902
... instead of truncating them.
There's no fixed limit for acceptable cookie names in RFC 6265, but the
entire cookie is said to be less than 4096 bytes (section 6.1). This is
also what browsers seem to implement.
We now allow max 5000 bytes cookie header. Max 4095 bytes length per
cookie name and value. Name + value together may not exceed 4096 bytes.
Added test 1151 to verify
Bug: https://curl.haxx.se/mail/lib-2017-09/0062.html
Reported-by: Kevin Smith
Closes#1894
The stub implementation is pre-loaded using LD_PRELOAD
and emulates common gssapi uses (only builds if curl is
initially built with gssapi support).
The initial tests are currently disabled for debug builds
as LD_PRELOAD is not used then.
Ref: https://github.com/curl/curl/pull/1687
Now that the form API is deprecated and not used anymore in curl tool,
a lot of its features left untested. Test 650 attempts to check all these
features not tested elsewhere.
curl_mime_encoder() is operational and documented.
curl tool -F option is extended with ";encoder=".
curl tool --libcurl option generates calls to curl_mime_encoder().
New encoder tests 648 & 649.
Test 1404 extended with an encoder specification.
This feature is badly supported in Windows: as a replacement, a caller has
to use curl_mime_data_cb() with fread, fseek and possibly fclose
callbacks to process opened files.
The cli tool and documentation are updated accordingly.
The feature is however kept internally for form API compatibility, with
the known caveats it always had.
As a side effect, stdin size is not determined by the cli tool even if
possible and this results in a chunked transfer encoding. Test 173 is
updated accordingly.
Additional mime-specific tests.
Existing tests updated to reflect small differences (Expect: 100-continue,
data size change due to empty lines, etc).
Option -F headers= keyword added to tests.
test1135 disabled until the entry point order change is resolved.
New example smtp-mime.
Examples postit2 and multi-post converted from form API to mime API.
Let's add a compile time safe API to select an SSL backend. This
function needs to be called *before* curl_global_init(), and can be
called only once.
Side note: we do not explicitly test that it is called before
curl_global_init(), but we do verify that it is not called multiple times
(even implicitly).
If SSL is used before the function was called, it will use whatever the
CURL_SSL_BACKEND environment variable says (or default to the first
available SSL backend), and if a subsequent call to
curl_global_sslset() disagrees with the previous choice, it will fail
with CURLSSLSET_TOO_LATE.
The function also accepts an "avail" parameter to point to a (read-only)
NULL-terminated list of available backends. This comes in real handy if
an application wants to let the user choose between whatever SSL backends
the currently available libcurl has to offer: simply call
curl_global_sslset(-1, NULL, &avail);
which will return CURLSSLSET_UNKNOWN_BACKEND and populate the avail
variable to point to the relevant information to present to the user.
Just like with the HTTP/2 push functions, we have to add the function
declaration of curl_global_sslset() function to the header file
*multi.h* because VMS and OS/400 require a stable order of functions
declared in include/curl/*.h (where the header files are sorted
alphabetically). This looks a bit funny, but it cannot be helped.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Fixes
$ valgrind --leak-check=full ~/install-curl-git/bin/curl tftp://localhost/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaz
==9752== Memcheck, a memory error detector
==9752== Copyright (C) 2002-2015, and GNU GPL'd, by Julian Seward et al.
==9752== Using Valgrind-3.11.0 and LibVEX; rerun with -h for copyright info
==9752== Command: /home/even/install-curl-git/bin/curl tftp://localhost/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaz
==9752==
curl: (71) TFTP file name too long
==9752==
==9752== HEAP SUMMARY:
==9752== 505 bytes in 1 blocks are definitely lost in loss record 11 of 11
==9752== at 0x4C2DB8F: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==9752== by 0x4E61CED: Curl_urldecode (in /home/even/install-curl-git/lib/libcurl.so.4.4.0)
==9752== by 0x4E75868: tftp_state_machine (in /home/even/install-curl-git/lib/libcurl.so.4.4.0)
==9752== by 0x4E761B6: tftp_do (in /home/even/install-curl-git/lib/libcurl.so.4.4.0)
==9752== by 0x4E711B6: multi_runsingle (in /home/even/install-curl-git/lib/libcurl.so.4.4.0)
==9752== by 0x4E71D00: curl_multi_perform (in /home/even/install-curl-git/lib/libcurl.so.4.4.0)
==9752== by 0x4E6950D: curl_easy_perform (in /home/even/install-curl-git/lib/libcurl.so.4.4.0)
==9752== by 0x40E0B7: operate_do (in /home/even/install-curl-git/bin/curl)
==9752== by 0x40E849: operate (in /home/even/install-curl-git/bin/curl)
==9752== by 0x402693: main (in /home/even/install-curl-git/bin/curl)
Fixes https://oss-fuzz.com/v2/testcase-detail/5232311106797568
Credit to OSS Fuzz
Closes#1808
The required low-level logic was already available as part of
`libssh2` (via `LIBSSH2_FLAG_COMPRESS` `libssh2_session_flag()`[1]
option.)
This patch adds the new `libcurl` option `CURLOPT_SSH_COMPRESSION`
(boolean) and the new `curl` command-line option `--compressed-ssh`
to request this `libssh2` feature. To have compression enabled, it
is required that the SSH server supports a (zlib) compatible
compression method and that `libssh2` was built with `zlib` support
enabled.
[1] https://www.libssh2.org/libssh2_session_flag.html
Ref: https://github.com/curl/curl/issues/1732
Closes https://github.com/curl/curl/pull/1735
... since CURLOPT_URL should follow the same rules as other options:
they remain set until changed or cleared.
Added test 1551 to verify.
Fixes#1631Closes#1632
Reported-by: Pavel Rochnyak
Prevent `Curl_pgrsTime` from modifying `t_starttransfer` when invoked
with `TIMER_STARTTRANSFER` more than once during a single request.
When a redirect occurs, this is considered a new request and
`t_starttransfer` can be updated to reflect the `t_starttransfer` time
of the redirect request.
Closes#1616
Bug: https://github.com/curl/curl/pull/1602#issuecomment-310267370
... to enable sending "OPTIONS *" which wasn't possible previously.
This option currently only works for HTTP.
Added test cases 1298 + 1299 to verify
Fixes#1280Closes#1462
The list was freed incorrectly since the llist refactor of
cbae73e1dd. Added test 1550 to verify that it works and avoid future
regressions.
Reported-by: Pascal Terjan
Fixes#1584Closes#1585
... unless "--output -" is used. Binary detection is done by simply
checking for a binary zero in early data.
Added test 1425 1426 to verify.
Closes#1512
Both these tests run the same underlying test code: libntlmconnect.c -
this test code made some assumptions about socket ordering when it used
curl_easy_fdset() and when we changed timing or got accidental changes
in libcurl the tests would fail.
The tests verify that the different transfers keep using the same
connections, which I now instead made sure by adding the number of bytes
each transfer gets and then verifies that they always get the same
amount as when these tests worked.
Closes#1576
mk-lib1521.pl generates a test program (lib1521.c) that calls
curl_easy_setopt() for every known option with a few typical values to
make sure they work (ignoring the return codes).
Some small changes were necessary to avoid asserts and NULL accesses
when doing this.
The perl script needs to be manually rerun when we add new options.
Closes#1543
Also removed a TODO suggesting caching the precheck results. Tests
showed this would save about 0.1 sec on the total test run time on a
relatively modern system, an unnoticeable gain at the cost of longer and
more complicated code. There would also be a danger that a cached test
result would be inappropriately returned, such as when other test
dependencies (like environment variables) are different or when the
precheck causes side effects (like filesystem changes).
@MarcelRaad noted that `test1399` causes infinite loop on MinGW.
Looking into this, seems like it is related to how Windows handles
CRLF. See https://github.com/curl/curl/commit/9e093f by @mback2k.
Removing `test1399` as it's identical to `test1326` then with such a
fix.
Test 1399 was broughy by commit 862b02f894Closes#1478
Include the test number in the names of files written out by tests to
reduce the chance of accidental duplication and to make it more clear
which test is associated with which file.
Windows does not allow setting the locale with environment variables (as
the test attempted to do), so the test failed when run with a user
locale that has a comma as radixchar. Changed the test to call
setlocale() explicitly to ensure that a known working locale is set even
on Windows.
Previous TODO wanting to write in chunks. We should support writing more
at once since some TELNET servers may respond immediately upon first
byte written such as WHOIS servers.
Closes#1389
Test command 'time curl http://localhost/80GB -so /dev/null' on a Debian
Linux.
Before (middle performing run out 9):
real 0m28.078s
user 0m11.240s
sys 0m12.876s
After (middle performing run out 9)
real 0m26.356s (93.9%)
user 0m5.324s (47.4%)
sys 0m8.368s (65.0%)
Also, doing SFTP over a 200 millsecond latency link is now about 6 times
faster.
Closes#1446
This checks the new behavior of Curl_splaygetbest, so that the smallest
node not larger than the key is removed, and FIFO behavior is kept even
when there are multiple nodes with the same key.
Closes#1358
system.h is aimed to replace curlbuild.h at a later point in time when
we feel confident system.h works sufficiently well.
curl/system.h is currently used in parallel with curl/curlbuild.h
curl/system.h determines a data sizes, data types and include file
status based on available preprocessor defines instead of getting
generated at build-time. This, in order to avoid relying on a build-time
generated file that makes it complicated to do 32 and 64 bit bields from
the same installed set of headers.
Test 1541 verifies that system.h comes to the same conclusion that
curlbuild.h offers.
Closes#1373
When receiving chunked encoded data with trailers, and the write
callback returns PAUSE, there might be both body and header to store to
resend on unpause. Previously libcurl returned error for that case.
Added test case 1540 to verify.
Reported-by: Stephen Toub
Fixes#1354Closes#1357
When using basic-auth, connections and proxy connections
can be re-used with different Authorization headers since
it does not authenticate the connection (like NTLM does).
For instance, the below command should re-use the proxy
connection, but it currently doesn't:
curl -v -U alice:a -x http://localhost:8181http://localhost/
--next -U bob:b -x http://localhost:8181http://localhost/
This is a regression since refactoring of ConnectionExists()
as part of: cb4e2be7c6
Fix the above by removing the username and password compare
when re-using proxy connection at proxy_info_matches().
However, this fix brings back another bug would make curl
to re-print the old proxy-authorization header of previous
proxy basic-auth connection because it wasn't cleared.
For instance, in the below command the second request should
fail if the proxy requires authentication, but would succeed
after the above fix (and before aforementioned commit):
curl -v -U alice:a -x http://localhost:8181http://localhost/
--next -x http://localhost:8181http://localhost/
Fix this by clearing conn->allocptr.proxyuserpwd after use
unconditionally, same as we do for conn->allocptr.userpwd.
Also fix test 540 to not expect digest auth header to be
resent when connection is reused.
Signed-off-by: Isaac Boukris <iboukris@gmail.com>
Closes https://github.com/curl/curl/pull/1350
Depend on the known behaviour of URLs for nonexistent files rather than
the undefined behaviour of URLs for directories (which fails on Windows).
The test isn't about file: URLs at all, so the URL used doesn't really
matter.
If a % ended the statement, the string's trailing NUL would be skipped
and memory past the end of the buffer would be accessed and potentially
displayed as part of the --write-out output. Added tests 1440 and 1441
to check for this kind of condition.
Reported-by: Brian Carpenter
- Add new option CURLOPT_SUPPRESS_CONNECT_HEADERS to allow suppressing
proxy CONNECT response headers from the user callback functions
CURLOPT_HEADERFUNCTION and CURLOPT_WRITEFUNCTION.
- Add new tool option --suppress-connect-headers to expose
CURLOPT_SUPPRESS_CONNECT_HEADERS and allow suppressing proxy CONNECT
response headers from --dump-header and --include.
Assisted-by: Jay Satiro
Assisted-by: CarloCannas@users.noreply.github.com
Closes https://github.com/curl/curl/pull/783
A client MUST ignore any Content-Length or Transfer-Encoding header
fields received in a successful response to CONNECT.
"Successful" described as: 2xx (Successful). RFC 7231 4.3.6
Prior to this change such a case would cause an error.
In some ways this bug appears to be a regression since c50b878. Prior to
that libcurl may have appeared to function correctly in such cases by
acting on those headers instead of causing an error. But that behavior
was also incorrect.
Bug: https://github.com/curl/curl/issues/1317
Reported-by: mkzero@users.noreply.github.com
Test 1903 is doing HTTP pipelining, and that is a timing and ordering
sensitive operation and this fails far too often on the Travis CI
leading to people more or less ignoring test failures there. Not good.
The end of pipelning is probably coming sooner rather than later
anyway...
... because it causes confusion with users. Example URLs:
"http://[127.0.0.1]:11211:80" which a lot of languages' URL parsers will
parse and claim uses port number 80, while libcurl would use port number
11211.
"http://user@example.com:80@localhost" which by the WHATWG URL spec will
be treated to contain user name 'user@example.com' but according to
RFC3986 is user name 'user' for the host 'example.com' and then port 80
is followed by "@localhost"
Both these formats are now rejected, and verified so in test 1260.
Reported-by: Orange Tsai
The character set in POSIX is set by the locale defined by (in
decreasing order of precedence) the LC_ALL, LC_CTYPE and LANG
environment variables (CHARSET was used by libidn but not libidn2).
LC_ALL is cleared to ensure that LC_CTYPE takes effect, but LC_ALL is
not used to set the locale to ensure that other parts of the locale
aren't overridden. Since there doesn't seem to be a cross-platform way
of specifying a UTF-8 locale, and not all systems may support UTF-8, a
<precheck> is used to skip the test if UTF-8 can't be verified to be
available. Test 1035 was also converted to UTF-8 for consistency, as
the actual character set used there is irrelevant to the test.
This patch uses a different UTF-8 locale than the last attempt, namely
en_US.UTF-8. This one has been verified on 7 different Linux and BSD
distributions and is more complete and usable than the locale UTF-8 (on
at least some systems).
This reverts commit ecd1d020ab.
That commit caused test failures on my Debian Linux machine for all
changed test cases. We need to reconsider how that should get done.
Character set in POSIX is set by the locale defined (in decreasing order
of precedence) by the LC_ALL, LC_CTYPE and LANG environment variables (I
believe CHARSET is only historic). LC_ALL is cleared to ensure that
LC_CTYPE takes effect, but LC_ALL is not used to set the locale to
ensure that other parts of the locale aren't overriden, if set. Since
there doesn't seem to be a cross-platform way of specifying a UTF-8
locale, and not all systems may support UTF-8, a <precheck> is used
(where relevant) to skip the test if UTF-8 isn't in use. Test 1035 was
also converted to UTF-8 for consistency, as the actual character set
used there is irrelevant to the test.
- on the first invocation: keep security context returned by
InitializeSecurityContext()
- on subsequent invocations: use MakeSignature() instead of
InitializeSecurityContext() to generate HTTP digest response
Bug: https://github.com/curl/curl/issues/870
Reported-by: Andreas Roth
Closes https://github.com/curl/curl/pull/1251
Properly resolve, convert and log the proxy host names.
Support the "--connect-to" feature for SOCKS proxies and for passive FTP
data transfers.
Follow-up to cb4e2be
Reported-by: Jay Satiro
Fixes https://github.com/curl/curl/issues/1248
- While negotiating auth during PUT/POST if a user-specified
Content-Length header is set send 'Content-Length: 0'.
This is what we do already in HTTPREQ_POST_FORM and what we did in the
HTTPREQ_POST case (regression since afd288b).
Prior to this change no Content-Length header would be sent in such a
case.
Bug: https://curl.haxx.se/mail/lib-2017-02/0006.html
Reported-by: Dominik Hölzl
Closes https://github.com/curl/curl/pull/1242
This repairs cookies for localhost.
Non-PSL builds will now only accept "localhost" without dots, while PSL
builds okeys everything not listed as PSL.
Added test 1258 to verify.
This was a regression brought in a76825a5ef
Under condition using http_proxy env var, noproxy list was the
combination of --noproxy option and NO_PROXY env var previously. Since
this commit, --noproxy option overrides NO_PROXY environment variable
even if use http_proxy env var.
Closes#1140
The combination of --noproxy option and http_proxy env var works well
both for proxied hosts and non-proxied hosts.
However, when combining NO_PROXY env var with --proxy option,
non-proxied hosts are not reachable while proxied host is OK.
This patch allows us to access non-proxied hosts even if using NO_PROXY
env var with --proxy option.
Follow-up to 3463408.
Prior to 3463408 file:// hostnames were silently stripped.
Prior to this commit it did not work when a schemeless url was used with
file as the default protocol.
Ref: https://curl.haxx.se/mail/lib-2016-11/0081.html
Closes https://github.com/curl/curl/pull/1124
Also fix for drive letters:
- Support --proto-default file c:/foo/bar.txt
- Support file://c:/foo/bar.txt
- Fail when a file:// drive letter is detected and not MSDOS/Windows.
Bug: https://github.com/curl/curl/issues/1187
Reported-by: Anatol Belski
Assisted-by: Anatol Belski
