In some situations, it was possible that a transfer was setup to
use an specific IP version, but due do DNS caching or connection
reuse, it ended up using a different IP version from requested.
This commit changes the effect of CURLOPT_IPRESOLVE from simply
restricting address resolution to preventing the wrong connection
type being used, when choosing a connection from the pool, and
to restricting what addresses could be used when establishing
a new connection.
It is important that all addresses versions are resolved, even if
not used in that transfer in particular, because the result is
cached, and could be useful for a different transfer with a
different CURLOPT_IPRESOLVE setting.
Closes#6853
Also added 'CURL_SMALLSENDS' to make Curl_write() send short packets,
which helped verifying this even more.
Add test 363 to verify.
Reported-by: ustcqidi on github
Fixes#6950Closes#7024
When hyper is used, it emits uppercase hexadecimal numbers for chunked
encoding lengths. Without hyper, lowercase hexadecimal numbers are used.
This change adds preprocessor statements to tests where this is an
issue, and adapts the fixtures to match.
Closes#6987
When a TLS server requests a client certificate during handshake and
none can be provided, libcurl now returns this new error code
CURLE_SSL_CLIENTCERT
Only supported by Secure Transport and OpenSSL for TLS 1.3 so far.
Closes#6721
A struct bufref holds a buffer pointer, a data size and a destructor.
When freed or its contents are changed, the previous buffer is implicitly
released by the associated destructor. The data size, although not used
internally, allows binary data support.
A unit test checks its handling methods: test 1661
Closes#6654
This reverts commit 1cba36d216.
CMake provides properties that can be set on a target to rename the
output artifact without changing the name of a target.
Ref: #6899
When the host name in a URL is given as an IPv4 numerical address, the
address can be specified with dotted numericals in four different ways:
a32, a.b24, a.b.c16 or a.b.c.d and each part can be specified in
decimal, octal (0-prefixed) or hexadecimal (0x-prefixed).
Instead of passing on the name as-is and leaving the handling to the
underlying name functions, which made them not work with c-ares but work
with getaddrinfo, this change now makes the curl URL API itself detect
and "normalize" host names specified as IPv4 numericals.
The WHATWG URL Spec says this is an okay way to specify a host name in a
URL. RFC 3896 does not allow them, but curl didn't prevent them before
and it seems other RFC 3896-using tools have not either. Host names used
like this are widely supported by other tools as well due to the
handling being done by getaddrinfo and friends.
I decided to add the functionality into the URL API itself so that all
users of these functions get the benefits, when for example wanting to
compare two URLs. Also, it makes curl built to use c-ares now support
them as well and make curl builds more consistent.
The normalization makes HTTPS and virtual hosted HTTP work fine even
when curl gets the address specified using one of the "obscure" formats.
Test 1560 is extended to verify.
Fixes#6863Closes#6871
Add test 676 to verify that setting CURLOPT_COOKIEFILE to NULL again clears
the cookiejar from memory.
Reported-by: Stefan Karpinski
Fixes#6889Closes#6891
According to Microsoft document MS-NLMP, current flags usage is not
accurate: flag NTLMFLAG_NEGOTIATE_NTLM2_KEY controls the use of
extended security in an NTLM authentication message and NTLM version 2
cannot be negotiated within the protocol.
The solution implemented here is: if the extended security flag is set,
prefer using NTLM version 2 (as a server featuring extended security
should also support version 2). If version 2 has been disabled at
compile time, use extended security.
Tests involving NTLM are adjusted to this new behavior.
Fixes#6813Closes#6849
This was previously defined by the obsolete AC_TYPE_SIGNAL macro,
which was removed in 2682e5f5. The deprecation text says
> Your code may safely assume C89 semantics that RETSIGTYPE is void.
So, remove it and just use void instead.
Closes#6861
After 957bc1881e686f9714c4e6a01bf33535091f0e21, we no longer compute an
expected_size for directories. This has the upshot that when we compare
even an empty Range with the available size, we fail.
This brings back the previous behaviour, which was to succeed, but with
empty content. This also removes the "Accept-ranges: bytes" header,
which is nonsensical on directories.
Adds test 3016
Fixes#6845Closes#6846
To make sure the Host: header and the URL provide the same authority
portion when sent to the proxy, strip the default port number from the
URL if one was provided.
Reported-by: Michael Brown
Fixes#6769Closes#6778
When asked to resume a download, libcurl will convert that to HTTP logic
and if then the entire file is already transferred it will result in a
416 response from the HTTP server. With CURLOPT_FAILONERRROR set in that
scenario, it should *not* lead to an error return.
Updated test 1156, added test 1273
Reported-by: Jonathan Watt
Fixes#6740Closes#6753
Introduce a new runtests.pl command option: -rm
For now only required and implemented for Windows.
Ignore stunnel logs due to long running processes.
Requires Sysinternals handle[64].exe to be on PATH.
Reviewed-by: Jay Satiro
Ref: #6058Closes#6179
- Increase the minimum number of spaces between the option and the
description from 1 to 2.
Before:
~~~
-u, --user <user:password> Server user and password
-A, --user-agent <name> Send User-Agent <name> to server
-v, --verbose Make the operation more talkative
-V, --version Show version number and quit
-w, --write-out <format> Use output FORMAT after completion
--xattr Store metadata in extended file attributes
~~~
After:
~~~
-u, --user <user:password> Server user and password
-A, --user-agent <name> Send User-Agent <name> to server
-v, --verbose Make the operation more talkative
-V, --version Show version number and quit
-w, --write-out <format> Use output FORMAT after completion
--xattr Store metadata in extended file attributes
~~~
Closes https://github.com/curl/curl/pull/6674
Previously only a single -b cookie parameter was supported with the last
one winning. This adds support for supplying multiple -b params to have
them serialized semicolon separated. Both cookiefiles and cookies can be
entered multiple times.
Closes#6649
Reviewed-by: Daniel Stenberg <daniel@haxx.se>
- add CURLINFO_REFERER libcurl option
- add --write-out '%{referer}' command-line option
- extend --xattr command-line option to fill user.xdg.referrer.url extended
attribute with the referrer (if there was any)
Closes#6591
Using the %NOLISTENPORT to trigger a connection failure is somewhat
"risky" (since it isn't guaranteed to not be listened to) and caused
occasional CI problems. This fix changes the infused error to be a more
reliable one but still verifies the --write-out functionality properly -
which is the purpose of this test.
Reported-by: Jay Satiro
Fixes#6621Closes#6623
HTTP auth "accidentally" worked before this cleanup since the code would
always overwrite the connection credentials with the credentials from
the most recent transfer and since HTTP auth is typically done first
thing, this has not been an issue. It was still wrong and subject to
possible race conditions or future breakage if the sequence of functions
would change.
The data.set.str[] strings MUST remain unmodified exactly as set by the
user, and the credentials to use internally are instead set/updated in
state.aptr.*
Added test 675 to verify different credentials used in two requests done
over a reused HTTP connection, which previously behaved wrongly.
Fixes#6542Closes#6545
Rename it to 'httpwant' and make a cloned field in the state struct as
well for run-time updates.
Also: refuse non-supported HTTP versions. Verified with test 129.
Closes#6585
and rename it from 'ftp_list_only' since it is also used for SSH and
POP3. The state is updated internally for 'type=D' FTP URLs.
Added test case 1570 to verify.
Closes#6578
... and make sure the code never updates 'set.prefer_ascii' as it breaks
handle reuse which should use the setting as the user specified it.
Added test 1569 to verify: it first makes an FTP transfer with ';type=A'
and then another without type on the same handle and the second should
then use binary. Previously, curl failed this.
Closes#6578
- Deduplicate the logic used by write-out and write-out json.
Rather than have separate writeLong, writeString, etc, logic for
each of write-out and write-out json instead have respective shared
functions that can output either format and a 'use_json' parameter to
indicate whether it is json that is output.
This will make it easier to maintain. Rather than have to go through
two sets of logic now we only have to go through one.
- Support write-out %{errormsg} and %{exitcode} in json.
- Clarify in the doc that %{exitcode} is the exit code of the transfer.
Prior to this change it just said "The numerical exitcode" which
implies it's the exit code of the tool, and it's not necessarily that.
Closes https://github.com/curl/curl/pull/6544
This is useful for tests containing HTML inside of <data> sections.
For <img> tags it's not uncommon to be longer than the previous
limit of 79 bytes.
An example of a previously problem-causing tag is:
<img src="http://config.privoxy.org/send-banner?type=auto" border="0" title="Killed-http://www.privoxy.org/images/privoxy.png-by-size" width="88" height="31">
which is needed for a Privoxy test for the banners-by-size filter.
Previously it caused server failures like:
12:29:05.786961 ====> Client connect
12:29:05.787116 accept_connection 3 returned 4
12:29:05.787194 accept_connection 3 returned 0
12:29:05.787285 Read 119 bytes
12:29:05.787345 Process 119 bytes request
12:29:05.787407 Got request: GET /banners-by-size/9 HTTP/1.1
12:29:05.787464 Requested test number 9 part 0
12:29:05.787686 getpart() failed with error: -2
12:29:05.787744 - request found to be complete (9)
12:29:05.787912 getpart() failed with error: -2
12:29:05.788048 Wrote request (119 bytes) input to log/server.input
12:29:05.788157 Send response test9 section <data>
12:29:05.788443 getpart() failed with error: -2
12:29:05.788498 instructed to close connection after server-reply
12:29:05.788550 ====> Client disconnect 0
12:29:05.871448 exit_signal_handler: 15
12:29:05.871714 signalled to die
12:29:05.872040 ========> IPv4 sws (port 21108 pid: 51758) exits with signal (15)
runtests.pl has lots of internal variables one might want to
change in certain situations, but adding a dedicated option
for every single one of them isn't practical.
Usage:
./runtests.pl -o TESTDIR=$privoxy_curl_test_dir -o HOSTIP=10.0.0.1 ...
- show the summarized test result in the last line of the report
- do not use $_ after mapping it to a named variable
Doing that makes the code harder to follow.
- log the restraints sorted by the number of their occurrences
- fix language when logging restraints that only occured once
- let runhttpserver() use $TESTDIR instead of $srcdir
... so it works if a non-default $TESTDIR is being used.
It can contain additional restraints for test numbers,
keywords and tools.
The idea is to let third parties like the Privoxy project
distribute an exclude file with their tarballs that specifies
which curl tests are not expected to work when using Privoxy
as a proxy, without having to fork the whole curl test suite.
The syntax could be changed to be extendable and maybe
more closely reflect the "curl test" syntax. Currently
it's a bunch of lines like these:
test:$TESTNUMBER:Reason why this test with number $TESTNUMBER should be skipped
keyword:$KEYWORD:Reason why tests whose keywords contain the $KEYWORD should be skipped
tool:$TOOL:Reason why tests with tools that contain $TOOL should be skipped
To specify multiple $TESTNUMBERs, $KEYWORDs and $TOOLs
on a single line, split them with commas.
This is useful to change the behaviour of the script without
having to modify the file itself, for example to use a custom
compareparts() function that ignores header differences that
are expected to occur when an external proxy is being used.
Such differences are proxy-specific and thus the modifications
should be maintained together with the proxy.
... that should be used when executing the tests.
The assumption is that the proxy is an HTTP proxy.
This option should be used together with -L to provide
a customized compareparts() version that knows which
proxy-specific header differences should be ignored.
This option doesn't work for all test types yet.
missing CRs and modified %hostip
lib556/test556: use a real HTTP version to make test reuse more convenient
make sure the weekday in Date headers matches the date
test61: replace stray "^M" (5e 4d) at the end of a cookie with a '^M' (0d)
Gets the test working with external proxies like Privoxy again.
Closes#6463
... as we ignore it anyway because servers don't report the correct size
and proftpd even blatantly returns a 550.
Updates a set of tests accordingly.
Reported-by: awesomenode on github
Fixes#6564Closes#6565
- Add support services without region and service prefixes in
the URL endpoint (ex. Min.IO, GCP, Yandex Cloud, Mail.Ru Cloud Solutions, etc)
by providing region and service parameters via aws-sigv4 option.
- Add [:region[:service]] suffix to aws-sigv4 option;
- Fix memory allocation errors.
- Refactor memory management.
- Use Curl_http_method instead() STRING_CUSTOMREQUEST.
- Refactor canonical headers generating.
- Remove repeated sha256_to_hex() usage.
- Add some docs fixes.
- Add some codestyle fixes.
- Add overloaded strndup() for debug - curl_dbg_strndup().
- Update tests.
Closes#6524
- Pass the very long request header via file instead of command line.
Prior to this change the 49k very long request header string was passed
via command line and on Windows that is too long so it was truncated and
the test would fail (specifically msys CI).
Closes https://github.com/curl/curl/pull/6516
... in most cases instead of 'struct connectdata *' but in some cases in
addition to.
- We mostly operate on transfers and not connections.
- We need the transfer handle to log, store data and more. Everything in
libcurl is driven by a transfer (the CURL * in the public API).
- This work clarifies and separates the transfers from the connections
better.
- We should avoid "conn->data". Since individual connections can be used
by many transfers when multiplexing, making sure that conn->data
points to the current and correct transfer at all times is difficult
and has been notoriously error-prone over the years. The goal is to
ultimately remove the conn->data pointer for this reason.
Closes#6425
... and as a practical side-effect, make sure that the
Curl_easyopts_check() function is asserted in debug builds, which we
want to detect mismatches between the options list in easyoptions.c and
the options in curl.h
Found-by: Gisle Vanem
Bug: 08e8455ddd (commitcomment-45991815)Closes#6461
... so that a function can first use MIMEPOST and then set it to NULL to
reset it back to a blank POST.
Added test 584 to verify the fix.
Reported-by: Christoph M. Becker
Fixes#6455Closes#6456
