Commit Graph

367 Commits

Author SHA1 Message Date
Daniel Stenberg 51c0ebcff2
http: deal with partial CONNECT sends
Also added 'CURL_SMALLSENDS' to make Curl_write() send short packets,
which helped verifying this even more.

Add test 363 to verify.

Reported-by: ustcqidi on github
Fixes #6950
Closes #7024
2021-05-08 10:49:16 +02:00
Gilles Vollant 77fc3859b2 SSL: support in-memory CA certs for some backends
- New options CURLOPT_CAINFO_BLOB and CURLOPT_PROXY_CAINFO_BLOB to
  specify in-memory PEM certificates for OpenSSL, Schannel (Windows)
  and Secure Transport (Apple) SSL backends.

Prior to this change PEM certificates could only be imported from a file
and not from memory.

Co-authored-by: moparisthebest@users.noreply.github.com

Ref: https://github.com/curl/curl/pull/4679
Ref: https://github.com/curl/curl/pull/5677
Ref: https://github.com/curl/curl/pull/6109

Closes https://github.com/curl/curl/pull/6662
2021-05-05 02:29:16 -04:00
Patrick Monnerat 34cf40321c
bufref: buffer reference support
A struct bufref holds a buffer pointer, a data size and a destructor.
When freed or its contents are changed, the previous buffer is implicitly
released by the associated destructor. The data size, although not used
internally, allows binary data support.

A unit test checks its handling methods: test 1661

Closes #6654
2021-04-22 09:05:53 +02:00
Daniel Stenberg 6fc805d0c1
test972: verify the json output with jsonlint
Make sure one of the azure jobs has jsonlint installed so that the test
runs there.

Ref: #6905
2021-04-17 11:03:40 +02:00
Daniel Stenberg 520bd5225c
cookie: CURLOPT_COOKIEFILE set to NULL switches off cookies
Add test 676 to verify that setting CURLOPT_COOKIEFILE to NULL again clears
the cookiejar from memory.

Reported-by: Stefan Karpinski
Fixes #6889
Closes #6891
2021-04-14 23:09:36 +02:00
Daniel Stenberg 605aa03ac1
http_proxy: only loop on 407 + close if we have credentials
... to fix the retry-loop.

Add test 718 to verify.

Reported-by: Daniel Kurečka
Fixes #6828
Closes #6850
2021-04-05 22:49:07 +02:00
Luke Granger-Brown 6d930d7306
file: support GETing directories again
After 957bc1881e686f9714c4e6a01bf33535091f0e21, we no longer compute an
expected_size for directories. This has the upshot that when we compare
even an empty Range with the available size, we fail.

This brings back the previous behaviour, which was to succeed, but with
empty content. This also removes the "Accept-ranges: bytes" header,
which is nonsensical on directories.

Adds test 3016
Fixes #6845
Closes #6846
2021-04-05 12:22:07 +02:00
Viktor Szakats 7214288898
transfer: strip credentials from the auto-referer header field
Added test 2081 to verify.

CVE-2021-22876

Bug: https://curl.se/docs/CVE-2021-22876.html
2021-03-28 23:19:55 +02:00
Daniel Stenberg 6d176bee59
http: make 416 not fail with resume + CURLOPT_FAILONERRROR
When asked to resume a download, libcurl will convert that to HTTP logic
and if then the entire file is already transferred it will result in a
416 response from the HTTP server. With CURLOPT_FAILONERRROR set in that
scenario, it should *not* lead to an error return.

Updated test 1156, added test 1273

Reported-by: Jonathan Watt
Fixes #6740
Closes #6753
2021-03-17 08:26:46 +01:00
Daniel Stenberg 7b2f0676c2
FTP: allow SIZE to fail when doing (resumed) upload
Added test 362 to verify.

Reported-by: Jordan Brown
Regression since 7ea2e1d0c5 (7.73.0)
Fixes #6715
Closes #6725
2021-03-11 16:13:11 +01:00
Daniel Stenberg 46620b9743
http: use credentials from transfer, not connection
HTTP auth "accidentally" worked before this cleanup since the code would
always overwrite the connection credentials with the credentials from
the most recent transfer and since HTTP auth is typically done first
thing, this has not been an issue. It was still wrong and subject to
possible race conditions or future breakage if the sequence of functions
would change.

The data.set.str[] strings MUST remain unmodified exactly as set by the
user, and the credentials to use internally are instead set/updated in
state.aptr.*

Added test 675 to verify different credentials used in two requests done
over a reused HTTP connection, which previously behaved wrongly.

Fixes #6542
Closes #6545
2021-02-13 22:36:15 +01:00
Daniel Stenberg 88dd1a8a11
urldata: don't touch data->set.httpversion at run-time
Rename it to 'httpwant' and make a cloned field in the state struct as
well for run-time updates.

Also: refuse non-supported HTTP versions. Verified with test 129.

Closes #6585
2021-02-12 08:13:37 +01:00
Daniel Stenberg 8a964cb217
curl: add --fail-with-body
Prevent both --fail and --fail-with-body on the same command line.

Verify with test 349, 360 and 361.

Closes #6449
2021-02-11 08:35:27 +01:00
Daniel Stenberg 528f71c2ec
ftp: add 'list_only' to the transfer state struct
and rename it from 'ftp_list_only' since it is also used for SSH and
POP3. The state is updated internally for 'type=D' FTP URLs.

Added test case 1570 to verify.

Closes #6578
2021-02-09 14:06:28 +01:00
Daniel Stenberg 115c9e27f5
ftp: add 'prefer_ascii' to the transfer state struct
... and make sure the code never updates 'set.prefer_ascii' as it breaks
handle reuse which should use the setting as the user specified it.

Added test 1569 to verify: it first makes an FTP transfer with ';type=A'
and then another without type on the same handle and the second should
then use binary. Previously, curl failed this.

Closes #6578
2021-02-09 14:06:28 +01:00
Stephan Szabo 1269c80af1 file: Support unicode urls on windows
Closes https://github.com/curl/curl/pull/6501
2021-02-09 02:45:51 -05:00
Fabian Keil 203dc00718
runtests.pl: add %TESTNUMBER variable to make copying tests more convenient 2021-02-09 08:16:40 +01:00
Fabian Keil 3f0bef2b53
runtests.pl: add an -E option to specify an exclude file
It can contain additional restraints for test numbers,
keywords and tools.

The idea is to let third parties like the Privoxy project
distribute an exclude file with their tarballs that specifies
which curl tests are not expected to work when using Privoxy
as a proxy, without having to fork the whole curl test suite.

The syntax could be changed to be extendable and maybe
more closely reflect the "curl test" syntax. Currently
it's a bunch of lines like these:

test:$TESTNUMBER:Reason why this test with number $TESTNUMBER should be skipped
keyword:$KEYWORD:Reason why tests whose keywords contain the $KEYWORD should be skipped
tool:$TOOL:Reason why tests with tools that contain $TOOL should be skipped

To specify multiple $TESTNUMBERs, $KEYWORDs and $TOOLs
on a single line, split them with commas.
2021-02-09 00:28:28 +01:00
Fabian Keil b47ee58fb7
runtests.pl: add -L parameter to require additional perl libraries
This is useful to change the behaviour of the script without
having to modify the file itself, for example to use a custom
compareparts() function that ignores header differences that
are expected to occur when an external proxy is being used.

Such differences are proxy-specific and thus the modifications
should be maintained together with the proxy.
2021-02-09 00:28:28 +01:00
Daniel Stenberg e32601e13a
ftp: avoid SIZE when asking for a TYPE A file
... as we ignore it anyway because servers don't report the correct size
and proftpd even blatantly returns a 550.

Updates a set of tests accordingly.

Reported-by: awesomenode on github
Fixes #6564
Closes #6565
2021-02-04 23:06:03 +01:00
Dmitry Wagin 796ce293de
http: improve AWS HTTP v4 Signature auth
- Add support services without region and service prefixes in
the URL endpoint (ex. Min.IO, GCP, Yandex Cloud, Mail.Ru Cloud Solutions, etc)
by providing region and service parameters via aws-sigv4 option.
- Add [:region[:service]] suffix to aws-sigv4 option;
- Fix memory allocation errors.
- Refactor memory management.
- Use Curl_http_method instead() STRING_CUSTOMREQUEST.
- Refactor canonical headers generating.
- Remove repeated sha256_to_hex() usage.
- Add some docs fixes.
- Add some codestyle fixes.
- Add overloaded strndup() for debug - curl_dbg_strndup().
- Update tests.

Closes #6524
2021-01-30 23:48:22 +01:00
Fabian Keil 0015e06151
http: get CURLOPT_REQUEST_TARGET working with a HTTP proxy
Added test 1613 to verify.

Closes #6490
2021-01-20 15:21:13 +01:00
Fabian Keil 6fcf22654e
Add test1181: Proxy request with --proxy-header "Connection: Keep-Alive" 2021-01-19 11:41:23 +01:00
Fabian Keil 3bccc182a2
Add test1180: Proxy request with -H "Proxy-Connection: Keep-Alive"
At the moment the test fails as curl sends two Proxy-Connection
headers.
2021-01-19 11:41:23 +01:00
Daniel Stenberg 11b2dd86d3
test410: verify HTTPS GET with a 49K request header
skip test 410 for mesalink in the CI as it otherwise hangs "forever"
2021-01-18 16:10:34 +01:00
Daniel Stenberg 30e7641d7d
test1918: verify curl_easy_option_by_name() and curl_easy_option_by_id()
... and as a practical side-effect, make sure that the
Curl_easyopts_check() function is asserted in debug builds, which we
want to detect mismatches between the options list in easyoptions.c and
the options in curl.h

Found-by: Gisle Vanem
Bug: 08e8455ddd (commitcomment-45991815)

Closes #6461
2021-01-15 17:03:29 +01:00
Daniel Stenberg 1123a0eb24
mime: make sure setting MIMEPOST to NULL resets properly
... so that a function can first use MIMEPOST and then set it to NULL to
reset it back to a blank POST.

Added test 584 to verify the fix.

Reported-by: Christoph M. Becker

Fixes #6455
Closes #6456
2021-01-15 08:33:27 +01:00
Daniel Stenberg ab525c059e
http: have CURLOPT_FAILONERROR fail after all headers
... so that Retry-After and other meta-content can still be used.

Added 1634 to verify. Adjusted test 194 and 281 since --fail now also
includes the header-terminating CRLF in the output before it exits.

Fixes #6408
Closes #6409
2021-01-06 15:14:40 +01:00
Daniel Stenberg 648712eec1
httpauth: make multi-request auth work with custom port
When doing HTTP authentication and a port number set with CURLOPT_PORT,
the code would previously have the URL's port number override as if it
had been a redirect to an absolute URL.

Added test 1568 to verify.

Reported-by: UrsusArctos on github
Fixes #6397
Closes #6400
2021-01-02 23:40:47 +01:00
Daniel Stenberg 58c4b8d636
test: verify new --write-out variables
Extended test 1029 and added 1188
2020-12-21 16:38:53 +01:00
Matthias Gatto ebdb5f23cc
tests: add very simple AWS HTTP v4 Signature test
Signed-off-by: Matthias Gatto <matthias.gatto@outscale.com>
2020-12-21 16:28:03 +01:00
Daniel Stenberg debf23eead
mqtt: handle POST/PUBLISH without a set POSTFIELDSIZE
Detected by OSS-Fuzz
Bug: https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=28735

Added test 1916 and 1917 to verify.

Closes #6338
2020-12-18 12:54:05 +01:00
Daniel Stenberg 3abfadfc19
test1198/9: add two mqtt publish tests without payload lengths
Closes #6335
2020-12-17 13:55:02 +01:00
Daniel Stenberg f24784f914
test1272: test gophers 2020-12-15 12:58:19 +01:00
Daniel Stenberg ee644bc8b3
test493: verify --hsts upgrade and that %{url_effective} reflects that
Closes #6175
2020-11-06 08:22:34 +01:00
Daniel Stenberg 4d2f800677
curl.se: new home
Closes #6172
2020-11-04 23:59:47 +01:00
Daniel Stenberg 2cfc4ed983
hsts: add read/write callbacks
- read/write callback options
- man pages for the 4 new setopts
- test 1915 verifies the callbacks

Closes #5896
2020-11-03 16:08:48 +01:00
Daniel Stenberg 7385610d0c
hsts: add support for Strict-Transport-Security
- enable in the build (configure)
- header parsing
- host name lookup
- unit tests for the above
- CI build
- CURL_VERSION_HSTS bit
- curl_version_info support
- curl -V output
- curl-config --features
- CURLOPT_HSTS_CTRL
- man page for CURLOPT_HSTS_CTRL
- curl --hsts (sets CURLOPT_HSTS_CTRL and works with --libcurl)
- man page for --hsts
- save cache to disk
- load cache from disk
- CURLOPT_HSTS
- man page for CURLOPT_HSTS
- added docs/HSTS.md
- fixed --version docs
- adjusted curl_easy_duphandle

Closes #5896
2020-11-03 16:08:42 +01:00
Daniel Stenberg 15997f6db5
test122[12]: remove these two tests
... and remove the objnames scripts they tested. They're not used for
anything anymore so testing them serves no purpose!

Reported-by: Marc Hörsken
Fixes #6080
Closes #6081
2020-10-15 07:56:19 +02:00
Daniel Stenberg ddecd28268
test1465: verify --libcurl with binary POST data 2020-10-02 09:35:38 +02:00
Daniel Stenberg 4a4c724599
ftp: make a 552 response return CURLE_REMOTE_DISK_FULL
Added test 348 to verify. Added a 'STOR' command to the test FTP
server to enable test 348. Documented the command in FILEFORMAT.md

Reported-by: Duncan Wilcox
Fixes #6016
Closes #6017
2020-09-26 13:37:15 +02:00
Daniel Stenberg 3e80895da7
test1297: verify GOT_NOTHING with http proxy tunnel 2020-09-22 10:54:42 +02:00
anio 0c1e767e83
tool_writeout: add new writeout variable, %{num_headers}
This variable gives the number of headers.

Closes #5947
2020-09-15 00:37:08 +02:00
Daniel Stenberg 7ea2e1d0c5
ftp: a 550 response to SIZE returns CURLE_REMOTE_FILE_NOT_FOUND
This is primarily interesting for cases where CURLOPT_NOBODY is set as
previously curl would not return an error for this case.

MDTM getting 550 now also returns this error (it returned
CURLE_FTP_COULDNT_RETR_FILE before) in order to unify return codes for
missing files across protocols and specific FTP commands.

libcurl already returns error on a 550 as a MDTM response (when
CURLOPT_FILETIME is set). If CURLOPT_NOBODY is not set, an error would
happen subsequently anyway since the RETR command would fail.

Add test 1913 and 1914 to verify. Updated several tests accordingly due
to the updated SIZE behavior.

Reported-by: Tomas Berger
Fixes #5953
Closes #5957
2020-09-14 12:53:12 +02:00
Daniel Stenberg 5c2d2d2ace
test434: test -K use in a single line without newline
Closes #5946
2020-09-14 08:32:49 +02:00
Daniel Stenberg 5fbc3ee520
tests: add test1912 to the dist
Follow-up to 70984ce1be
2020-09-12 18:09:49 +02:00
Daniel Stenberg 8ca54a03ea
test1541: remove since it is a known bug
A shared connection cache is not thread-safe is a known issue. Stop
testing this until we believe this issue is addressed. Reduces
occasional test failures we don't care about.

The test code in lib1541.c is left in git to allow us to restore it when
we get to fix this.

Closes #5922
2020-09-06 09:51:06 +02:00
Daniel Stenberg e50a877df7
tests: remove pipelining tests
Remove the tests 530, 584, 1900, 1901, 1902, 1903 and 2033. They were
previously disabled.

The Pipelining code was removed from curl in commit 2f44e94efb,
April 2019.

Closes #5921
2020-09-06 09:48:17 +02:00
Emil Engler a86cc7e2ab
tests: Add tests for new --help
This commit is a part of "--help me if you can"

Closes #5680
2020-09-04 15:38:22 +02:00
Daniel Stenberg 6ebe63fac2
options: API for meta-data about easy options
const struct curl_easyoption *curl_easy_option_by_name(const char *name);

 const struct curl_easyoption *curl_easy_option_by_id (CURLoption id);

 const struct curl_easyoption *
 curl_easy_option_next(const struct curl_easyoption *prev);

The purpose is to provide detailed enough information to allow for
example libcurl bindings to get option information at run-time about
what easy options that exist and what arguments they expect.

Assisted-by: Jeroen Ooms
Closes #5365
2020-08-27 14:17:36 +02:00