The required low-level logic was already available as part of
`libssh2` (via `LIBSSH2_FLAG_COMPRESS` `libssh2_session_flag()`[1]
option.)
This patch adds the new `libcurl` option `CURLOPT_SSH_COMPRESSION`
(boolean) and the new `curl` command-line option `--compressed-ssh`
to request this `libssh2` feature. To have compression enabled, it
is required that the SSH server supports a (zlib) compatible
compression method and that `libssh2` was built with `zlib` support
enabled.
[1] https://www.libssh2.org/libssh2_session_flag.html
Ref: https://github.com/curl/curl/issues/1732
Closes https://github.com/curl/curl/pull/1735
... since CURLOPT_URL should follow the same rules as other options:
they remain set until changed or cleared.
Added test 1551 to verify.
Fixes#1631Closes#1632
Reported-by: Pavel Rochnyak
Prevent `Curl_pgrsTime` from modifying `t_starttransfer` when invoked
with `TIMER_STARTTRANSFER` more than once during a single request.
When a redirect occurs, this is considered a new request and
`t_starttransfer` can be updated to reflect the `t_starttransfer` time
of the redirect request.
Closes#1616
Bug: https://github.com/curl/curl/pull/1602#issuecomment-310267370
... to enable sending "OPTIONS *" which wasn't possible previously.
This option currently only works for HTTP.
Added test cases 1298 + 1299 to verify
Fixes#1280Closes#1462
The list was freed incorrectly since the llist refactor of
cbae73e1dd. Added test 1550 to verify that it works and avoid future
regressions.
Reported-by: Pascal Terjan
Fixes#1584Closes#1585
... unless "--output -" is used. Binary detection is done by simply
checking for a binary zero in early data.
Added test 1425 1426 to verify.
Closes#1512
Both these tests run the same underlying test code: libntlmconnect.c -
this test code made some assumptions about socket ordering when it used
curl_easy_fdset() and when we changed timing or got accidental changes
in libcurl the tests would fail.
The tests verify that the different transfers keep using the same
connections, which I now instead made sure by adding the number of bytes
each transfer gets and then verifies that they always get the same
amount as when these tests worked.
Closes#1576
mk-lib1521.pl generates a test program (lib1521.c) that calls
curl_easy_setopt() for every known option with a few typical values to
make sure they work (ignoring the return codes).
Some small changes were necessary to avoid asserts and NULL accesses
when doing this.
The perl script needs to be manually rerun when we add new options.
Closes#1543
Also removed a TODO suggesting caching the precheck results. Tests
showed this would save about 0.1 sec on the total test run time on a
relatively modern system, an unnoticeable gain at the cost of longer and
more complicated code. There would also be a danger that a cached test
result would be inappropriately returned, such as when other test
dependencies (like environment variables) are different or when the
precheck causes side effects (like filesystem changes).
@MarcelRaad noted that `test1399` causes infinite loop on MinGW.
Looking into this, seems like it is related to how Windows handles
CRLF. See https://github.com/curl/curl/commit/9e093f by @mback2k.
Removing `test1399` as it's identical to `test1326` then with such a
fix.
Test 1399 was broughy by commit 862b02f894Closes#1478
Include the test number in the names of files written out by tests to
reduce the chance of accidental duplication and to make it more clear
which test is associated with which file.
Windows does not allow setting the locale with environment variables (as
the test attempted to do), so the test failed when run with a user
locale that has a comma as radixchar. Changed the test to call
setlocale() explicitly to ensure that a known working locale is set even
on Windows.
Previous TODO wanting to write in chunks. We should support writing more
at once since some TELNET servers may respond immediately upon first
byte written such as WHOIS servers.
Closes#1389
Test command 'time curl http://localhost/80GB -so /dev/null' on a Debian
Linux.
Before (middle performing run out 9):
real 0m28.078s
user 0m11.240s
sys 0m12.876s
After (middle performing run out 9)
real 0m26.356s (93.9%)
user 0m5.324s (47.4%)
sys 0m8.368s (65.0%)
Also, doing SFTP over a 200 millsecond latency link is now about 6 times
faster.
Closes#1446
This checks the new behavior of Curl_splaygetbest, so that the smallest
node not larger than the key is removed, and FIFO behavior is kept even
when there are multiple nodes with the same key.
Closes#1358
system.h is aimed to replace curlbuild.h at a later point in time when
we feel confident system.h works sufficiently well.
curl/system.h is currently used in parallel with curl/curlbuild.h
curl/system.h determines a data sizes, data types and include file
status based on available preprocessor defines instead of getting
generated at build-time. This, in order to avoid relying on a build-time
generated file that makes it complicated to do 32 and 64 bit bields from
the same installed set of headers.
Test 1541 verifies that system.h comes to the same conclusion that
curlbuild.h offers.
Closes#1373
When receiving chunked encoded data with trailers, and the write
callback returns PAUSE, there might be both body and header to store to
resend on unpause. Previously libcurl returned error for that case.
Added test case 1540 to verify.
Reported-by: Stephen Toub
Fixes#1354Closes#1357
When using basic-auth, connections and proxy connections
can be re-used with different Authorization headers since
it does not authenticate the connection (like NTLM does).
For instance, the below command should re-use the proxy
connection, but it currently doesn't:
curl -v -U alice:a -x http://localhost:8181http://localhost/
--next -U bob:b -x http://localhost:8181http://localhost/
This is a regression since refactoring of ConnectionExists()
as part of: cb4e2be7c6
Fix the above by removing the username and password compare
when re-using proxy connection at proxy_info_matches().
However, this fix brings back another bug would make curl
to re-print the old proxy-authorization header of previous
proxy basic-auth connection because it wasn't cleared.
For instance, in the below command the second request should
fail if the proxy requires authentication, but would succeed
after the above fix (and before aforementioned commit):
curl -v -U alice:a -x http://localhost:8181http://localhost/
--next -x http://localhost:8181http://localhost/
Fix this by clearing conn->allocptr.proxyuserpwd after use
unconditionally, same as we do for conn->allocptr.userpwd.
Also fix test 540 to not expect digest auth header to be
resent when connection is reused.
Signed-off-by: Isaac Boukris <iboukris@gmail.com>
Closes https://github.com/curl/curl/pull/1350
Depend on the known behaviour of URLs for nonexistent files rather than
the undefined behaviour of URLs for directories (which fails on Windows).
The test isn't about file: URLs at all, so the URL used doesn't really
matter.
If a % ended the statement, the string's trailing NUL would be skipped
and memory past the end of the buffer would be accessed and potentially
displayed as part of the --write-out output. Added tests 1440 and 1441
to check for this kind of condition.
Reported-by: Brian Carpenter
- Add new option CURLOPT_SUPPRESS_CONNECT_HEADERS to allow suppressing
proxy CONNECT response headers from the user callback functions
CURLOPT_HEADERFUNCTION and CURLOPT_WRITEFUNCTION.
- Add new tool option --suppress-connect-headers to expose
CURLOPT_SUPPRESS_CONNECT_HEADERS and allow suppressing proxy CONNECT
response headers from --dump-header and --include.
Assisted-by: Jay Satiro
Assisted-by: CarloCannas@users.noreply.github.com
Closes https://github.com/curl/curl/pull/783
A client MUST ignore any Content-Length or Transfer-Encoding header
fields received in a successful response to CONNECT.
"Successful" described as: 2xx (Successful). RFC 7231 4.3.6
Prior to this change such a case would cause an error.
In some ways this bug appears to be a regression since c50b878. Prior to
that libcurl may have appeared to function correctly in such cases by
acting on those headers instead of causing an error. But that behavior
was also incorrect.
Bug: https://github.com/curl/curl/issues/1317
Reported-by: mkzero@users.noreply.github.com
Test 1903 is doing HTTP pipelining, and that is a timing and ordering
sensitive operation and this fails far too often on the Travis CI
leading to people more or less ignoring test failures there. Not good.
The end of pipelning is probably coming sooner rather than later
anyway...
... because it causes confusion with users. Example URLs:
"http://[127.0.0.1]:11211:80" which a lot of languages' URL parsers will
parse and claim uses port number 80, while libcurl would use port number
11211.
"http://user@example.com:80@localhost" which by the WHATWG URL spec will
be treated to contain user name 'user@example.com' but according to
RFC3986 is user name 'user' for the host 'example.com' and then port 80
is followed by "@localhost"
Both these formats are now rejected, and verified so in test 1260.
Reported-by: Orange Tsai
The character set in POSIX is set by the locale defined by (in
decreasing order of precedence) the LC_ALL, LC_CTYPE and LANG
environment variables (CHARSET was used by libidn but not libidn2).
LC_ALL is cleared to ensure that LC_CTYPE takes effect, but LC_ALL is
not used to set the locale to ensure that other parts of the locale
aren't overridden. Since there doesn't seem to be a cross-platform way
of specifying a UTF-8 locale, and not all systems may support UTF-8, a
<precheck> is used to skip the test if UTF-8 can't be verified to be
available. Test 1035 was also converted to UTF-8 for consistency, as
the actual character set used there is irrelevant to the test.
This patch uses a different UTF-8 locale than the last attempt, namely
en_US.UTF-8. This one has been verified on 7 different Linux and BSD
distributions and is more complete and usable than the locale UTF-8 (on
at least some systems).
This reverts commit ecd1d020ab.
That commit caused test failures on my Debian Linux machine for all
changed test cases. We need to reconsider how that should get done.
Character set in POSIX is set by the locale defined (in decreasing order
of precedence) by the LC_ALL, LC_CTYPE and LANG environment variables (I
believe CHARSET is only historic). LC_ALL is cleared to ensure that
LC_CTYPE takes effect, but LC_ALL is not used to set the locale to
ensure that other parts of the locale aren't overriden, if set. Since
there doesn't seem to be a cross-platform way of specifying a UTF-8
locale, and not all systems may support UTF-8, a <precheck> is used
(where relevant) to skip the test if UTF-8 isn't in use. Test 1035 was
also converted to UTF-8 for consistency, as the actual character set
used there is irrelevant to the test.
- on the first invocation: keep security context returned by
InitializeSecurityContext()
- on subsequent invocations: use MakeSignature() instead of
InitializeSecurityContext() to generate HTTP digest response
Bug: https://github.com/curl/curl/issues/870
Reported-by: Andreas Roth
Closes https://github.com/curl/curl/pull/1251
Properly resolve, convert and log the proxy host names.
Support the "--connect-to" feature for SOCKS proxies and for passive FTP
data transfers.
Follow-up to cb4e2be
Reported-by: Jay Satiro
Fixes https://github.com/curl/curl/issues/1248
- While negotiating auth during PUT/POST if a user-specified
Content-Length header is set send 'Content-Length: 0'.
This is what we do already in HTTPREQ_POST_FORM and what we did in the
HTTPREQ_POST case (regression since afd288b).
Prior to this change no Content-Length header would be sent in such a
case.
Bug: https://curl.haxx.se/mail/lib-2017-02/0006.html
Reported-by: Dominik Hölzl
Closes https://github.com/curl/curl/pull/1242
This repairs cookies for localhost.
Non-PSL builds will now only accept "localhost" without dots, while PSL
builds okeys everything not listed as PSL.
Added test 1258 to verify.
This was a regression brought in a76825a5ef
Under condition using http_proxy env var, noproxy list was the
combination of --noproxy option and NO_PROXY env var previously. Since
this commit, --noproxy option overrides NO_PROXY environment variable
even if use http_proxy env var.
Closes#1140
The combination of --noproxy option and http_proxy env var works well
both for proxied hosts and non-proxied hosts.
However, when combining NO_PROXY env var with --proxy option,
non-proxied hosts are not reachable while proxied host is OK.
This patch allows us to access non-proxied hosts even if using NO_PROXY
env var with --proxy option.
Follow-up to 3463408.
Prior to 3463408 file:// hostnames were silently stripped.
Prior to this commit it did not work when a schemeless url was used with
file as the default protocol.
Ref: https://curl.haxx.se/mail/lib-2016-11/0081.html
Closes https://github.com/curl/curl/pull/1124
Also fix for drive letters:
- Support --proto-default file c:/foo/bar.txt
- Support file://c:/foo/bar.txt
- Fail when a file:// drive letter is detected and not MSDOS/Windows.
Bug: https://github.com/curl/curl/issues/1187
Reported-by: Anatol Belski
Assisted-by: Anatol Belski
Follow-up to 82245ea: Fix the example program sendrecv.c (handle
CURLE_AGAIN, handle incomplete send). Improve the documentation
for curl_easy_recv() and curl_easy_send().
Reviewed-by: Frank Meier
Assisted-by: Jay Satiro
See https://github.com/curl/curl/pull/1134
A server MUST NOT send any Transfer-Encoding or Content-Length header
fields in a 2xx (Successful) response to CONNECT. (RFC 7231 section
4.3.6)
Also fixes the three test cases that did this.
If a port number in a "connect-to" entry does not match, skip this
entry instead of connecting to port 0.
If a port number in a "connect-to" entry matches, use this entry
and look no further.
Reported-by: Jay Satiro
Assisted-by: Jay Satiro, Daniel Stenberg
Closes#1148
We're mostly saying just "curl" in lower case these days so here's a big
cleanup to adapt to this reality. A few instances are left as the
project could still formally be considered called cURL.
- Call Curl_initinfo on init and duphandle.
Prior to this change the statistical and informational variables were
simply zeroed by calloc on easy init and duphandle. While zero is the
correct default value for almost all info variables, there is one where
it isn't (filetime initializes to -1).
Bug: https://github.com/curl/curl/issues/1103
Reported-by: Neal Poole
Cokie with the same domain but different tailmatching property are now
considered different and do not replace each other. If header contains
following lines then two cookies will be set: Set-Cookie: foo=bar;
domain=.foo.com; expires=Thu Mar 3 GMT 8:56:27 2033 Set-Cookie: foo=baz;
domain=foo.com; expires=Thu Mar 3 GMT 8:56:27 2033
This matches Chrome, Opera, Safari, and Firefox behavior. When sending
stored tokens to foo.com Chrome, Opera, Firefox store send them in the
stored order, while Safari pre-sort the cookies.
Closes#1050
Add the new option CURLOPT_KEEP_SENDING_ON_ERROR to control whether
sending the request body shall be completed when the server responds
early with an error status code.
This is suitable for manual NTLM authentication.
Reviewed-by: Jay Satiro
Closes https://github.com/curl/curl/pull/904
.. and add that --proto-redir and CURLOPT_REDIR_PROTOCOLS do not
override protocols denied by --proto and CURLOPT_PROTOCOLS.
- Add a test to enforce: --proto deny must override --proto-redir allow
Closes https://github.com/curl/curl/pull/1031
... like when a HTTP/0.9 response comes back without any headers at all
and just a body this now prevents that body from being sent to the
callback etc.
Adapted test 1144 to verify.
Fixes#973
Assisted-by: Ray Satiro
Since we're using CURLE_FTP_WEIRD_SERVER_REPLY in imap, pop3 and smtp as
more of a generic "failed to parse" introduce an alias without FTP in
the name.
Closes https://github.com/curl/curl/pull/975
With HTTP/2 each transfer is made in an indivial logical stream over the
connection, making most previous errors that caused the connection to get
forced-closed now instead just kill the stream and not the connection.
Fixes#941
This fixes tests that were added after 113f04e664 as the tests would
fail otherwise.
We bring back "Proxy-Connection: Keep-Alive" now unconditionally to fix
regressions with old and stupid proxies, but we could possibly switch to
using it only for CONNECT or only for NTLM in a future if we want to
gradually reduce it.
Fixes#954
Reported-by: János Fekete
The HTTP/2 tests brought with commit bf05606ef1 were using the internal
name 'http2' for the HTTP/2 server, while in fact that name was already
used for the second instance of the HTTP server. This made tests using
the second instance (like test 2050) fail after a HTTP/2 test had run.
The server is now known as HTTP/2 internally and within the <server>
section in test cases. 1700, 1701 and 1702 were updated accordingly.
It requires that 'nghttpx' is in the PATH, and it will run the tests
using nghttpx as a front-end proxy in front of the standard HTTP/1 test
server. This uses HTTP/2 over plain TCP.
If you like me have nghttpx installed in a custom path, you can run test 1700
like this:
$ PATH=$PATH:$HOME/build-nghttp2/bin/ ./runtests.pl 1700
Mostly in order to support broken web sites that redirect to broken URLs
that are accepted by browsers.
Browsers are typically even more leniant than this as the WHATWG URL
spec they should allow an _infinite_ amount. I tested 8000 slashes with
Firefox and it just worked.
Added test case 1141, 1142 and 1143 to verify the new parser.
Closes#791
Prior to this change a width arg could be erroneously output, and also
width and precision args could not be used together without crashing.
"%0*d%s", 2, 9, "foo"
Before: "092"
After: "09foo"
"%*.*s", 5, 2, "foo"
Before: crash
After: " fo"
Test 557 is updated to verify this and more
It does open up a miniscule risk that one of the other protocols that
libcurl could use would send back a Content-Disposition header and then
curl would act on it even if not HTTP.
A future mitigation for this risk would be to allow the callback to ask
libcurl which protocol is being used.
Verified with test 1312
Closes#760
While being debated (in #716) and a violation of RFC 7230 section 5.4,
this test verifies that the existing functionality works as intended. It
strips the dot from the host name and uses the host without dot
throughout the internals.
... for checking ability to receive full HTTP response when POST request
is used with slow read callback function.
This test checks for bug #657 and verifies the work-around from
72d5e144fb.
Closes#720
RFC 6265 section 4.1.1 spells out that the first name/value pair in the
header is the actual cookie name and content, while the following are
the parameters.
libcurl previously had a more liberal approach which causes significant
problems when introducing new cookie parameters, like the suggested new
cookie priority draft.
The previous logic read all n/v pairs from left-to-right and the first
name used that wassn't a known parameter name would be used as the
cookie name, thus accepting "Set-Cookie: Max-Age=2; person=daniel" to be
a cookie named 'person' while an RFC 6265 compliant parser should
consider that to be a cookie named 'Max-Age' with an (unknown) parameter
'person'.
Fixes#709
- Add tests.
- Add an example to CURLOPT_TFTP_NO_OPTIONS.3.
- Add --tftp-no-options to expose CURLOPT_TFTP_NO_OPTIONS.
Bug: https://github.com/curl/curl/issues/481
It turns out Firefox and Chrome both allow spaces in cookie names and
there are sites out there using that.
Turned out the code meant to strip off trailing space from cookie names
didn't work. Fixed now.
Test case 8 modified to verify both these changes.
Closes#639
- Add unit test 1604 to test the sanitize_file_name function.
- Use -DCURL_STATICLIB when building libcurltool for unit testing.
- Better detection of reserved DOS device names.
- New flags to modify sanitize behavior:
SANITIZE_ALLOW_COLONS: Allow colons
SANITIZE_ALLOW_PATH: Allow path separators and colons
SANITIZE_ALLOW_RESERVED: Allow reserved device names
SANITIZE_ALLOW_TRUNCATE: Allow truncating a long filename
- Restore sanitization of banned characters from user-specified outfile.
Prior to this commit sanitization of a user-specified outfile was
temporarily disabled in 2b6dadc because there was no way to allow path
separators and colons through while replacing other banned characters.
Now in such a case we call the sanitize function with
SANITIZE_ALLOW_PATH which allows path separators and colons to pass
through.
Closes https://github.com/curl/curl/issues/624
Reported-by: Octavio Schroeder
Before this patch, if a URL does not start with the protocol
name/scheme, effective URLs would be prefixed with upper-case protocol
names/schemes. This behavior might not be expected by library users or
end users.
For example, if `CURLOPT_DEFAULT_PROTOCOL` is set to "https". And the
URL is "hostname/path". The effective URL would be
"HTTPS://hostname/path" instead of "https://hostname/path".
After this patch, effective URLs would be prefixed with a lower-case
protocol name/scheme.
Closes#597
Signed-off-by: Mohammad AlSaleh <CE.Mohammad.AlSaleh@gmail.com>
The request needs to be read and send in binary mode in order to use
CRLF instead of LF. Adding --upload-file - causes curl to read stdin
in binary mode.
Make this the default for the curl tool (if built with HTTP/2 powers
enabled) unless a specific HTTP version is requested on the command
line.
This should allow more users to get HTTP/2 powers without having to
change anything.
Tests 842, 843, 844, 845, 887, 888, 889, 890, 946, 947, 948 and 949 fail
if a custom port number is specified via the -b option of runtests.pl.
Suggested by: Kamil Dudka
Bug: http://curl.haxx.se/mail/lib-2015-12/0003.html
As POP3 final and continuation responses both begin with a + character,
and both the finalcode and contcode variables in SASLprotoc are set as
such, we cannot tell the difference between them when we are expecting
an optional continuation from the server such as the following:
+ something else from the server
+OK final response
Disabled these tests until such a time we can tell the responses apart.
The tftpd test server now logs all received options and thus all TFTP
test cases need to match them exactly.
Extended test 283 to use and verify --tftp-blksize.
Apparently there are sites out there that do redirects to URLs they
provide in plain UTF-8 or similar. Browsers and wget %-encode such
headers when doing a subsequent request. Now libcurl does too.
Added test 1138 to verify.
Closes#473
