... and as a practical side-effect, make sure that the
Curl_easyopts_check() function is asserted in debug builds, which we
want to detect mismatches between the options list in easyoptions.c and
the options in curl.h
Found-by: Gisle Vanem
Bug: 08e8455ddd (commitcomment-45991815)Closes#6461
... so that a function can first use MIMEPOST and then set it to NULL to
reset it back to a blank POST.
Added test 584 to verify the fix.
Reported-by: Christoph M. Becker
Fixes#6455Closes#6456
... so that Retry-After and other meta-content can still be used.
Added 1634 to verify. Adjusted test 194 and 281 since --fail now also
includes the header-terminating CRLF in the output before it exits.
Fixes#6408Closes#6409
When doing HTTP authentication and a port number set with CURLOPT_PORT,
the code would previously have the URL's port number override as if it
had been a redirect to an absolute URL.
Added test 1568 to verify.
Reported-by: UrsusArctos on github
Fixes#6397Closes#6400
This option sets the (octal) mode to use for the remote file when one is
created, using the SFTP, SCP or FILE protocols. When not set, the
default is 0644.
Closes#6244
When the initial request isn't possible to send in its entirety, the
remainder of request would be delivered to the debug callback as data
and would wrongly be counted internally as body-bytes sent.
Extended test 1295 to verify.
Closes#6328
The command line tool also independently sets --ftp-skip-pasv-ip by
default.
Ten test cases updated to adapt the modified --libcurl output.
Bug: https://curl.se/docs/CVE-2020-8284.html
CVE-2020-8284
Reported-by: Varnavas Papaioannou
... as it makes the URL parser accept "very-long-hostname://" as a valid
host name and we don't want that. The parser now only accepts a blank
(no digits) after the colon if the URL starts with a scheme.
Reported-by: d4d on hackerone
Closes#6283
- enable in the build (configure)
- header parsing
- host name lookup
- unit tests for the above
- CI build
- CURL_VERSION_HSTS bit
- curl_version_info support
- curl -V output
- curl-config --features
- CURLOPT_HSTS_CTRL
- man page for CURLOPT_HSTS_CTRL
- curl --hsts (sets CURLOPT_HSTS_CTRL and works with --libcurl)
- man page for --hsts
- save cache to disk
- load cache from disk
- CURLOPT_HSTS
- man page for CURLOPT_HSTS
- added docs/HSTS.md
- fixed --version docs
- adjusted curl_easy_duphandle
Closes#5896
- Add regex that strips http/2 server header name to those http/2 tests
that don't already have it.
- Improve that regex in all http/2 tests.
Tests 358 and 359 were failing for me before this change on a system
that uses an older version of nghttpx which includes its version number
in the server header.
Closes https://github.com/curl/curl/pull/6139
... and remove the objnames scripts they tested. They're not used for
anything anymore so testing them serves no purpose!
Reported-by: Marc Hörsken
Fixes#6080Closes#6081
... and use this new keywords in all the test files larger than 50K to reduce
their sizes and make them a lot easier to read and understand.
Closes#6040
Added test 348 to verify. Added a 'STOR' command to the test FTP
server to enable test 348. Documented the command in FILEFORMAT.md
Reported-by: Duncan Wilcox
Fixes#6016Closes#6017
When using HTTPS proxy, SSL is used but not in the view of the FTP
protocol handler itself so separate the connection's use of SSL from the
FTP control connection's sue.
Reported-by: Mingtao Yang
Fixes#5523Closes#6006
This is primarily interesting for cases where CURLOPT_NOBODY is set as
previously curl would not return an error for this case.
MDTM getting 550 now also returns this error (it returned
CURLE_FTP_COULDNT_RETR_FILE before) in order to unify return codes for
missing files across protocols and specific FTP commands.
libcurl already returns error on a 550 as a MDTM response (when
CURLOPT_FILETIME is set). If CURLOPT_NOBODY is not set, an error would
happen subsequently anyway since the RETR command would fail.
Add test 1913 and 1914 to verify. Updated several tests accordingly due
to the updated SIZE behavior.
Reported-by: Tomas Berger
Fixes#5953Closes#5957
The cache content is not duplicated, like other caches, but the setting
and specified file name are.
Test 1908 is extended to verify this somewhat. Since the duplicated
handle gets the same file name, the test unfortunately overwrites the
same file twice (with different contents) which makes it hard to check
automatically.
Closes#5923
A shared connection cache is not thread-safe is a known issue. Stop
testing this until we believe this issue is addressed. Reduces
occasional test failures we don't care about.
The test code in lib1541.c is left in git to allow us to restore it when
we get to fix this.
Closes#5922
Remove the tests 530, 584, 1900, 1901, 1902, 1903 and 2033. They were
previously disabled.
The Pipelining code was removed from curl in commit 2f44e94efb,
April 2019.
Closes#5921
const struct curl_easyoption *curl_easy_option_by_name(const char *name);
const struct curl_easyoption *curl_easy_option_by_id (CURLoption id);
const struct curl_easyoption *
curl_easy_option_next(const struct curl_easyoption *prev);
The purpose is to provide detailed enough information to allow for
example libcurl bindings to get option information at run-time about
what easy options that exist and what arguments they expect.
Assisted-by: Jeroen Ooms
Closes#5365
Failures clearly returned from a (SOCKS) proxy now causes this return
code. Previously the situation was not very clear as what would be
returned and when.
In addition: when this error code is returned, an application can use
CURLINFO_PROXY_ERROR to query libcurl for the detailed error, which then
returns a value from the new 'CURLproxycode' enum.
Closes#5770
The CMake option is now called CMAKE_USE_SCHANNEL
The winbuild flag is USE_SCHANNEL
The CI jobs and build scripts only use the new names and the new name
options
Tests now require 'Schannel' (when necessary)
Closes#5795
This allows us to make test cases to use base64 at run-time and still
use and verify information determined at run-time, such as the IMAP test
server's port number in test 842.
This change makes 12 tests run again that basically never ran since we
moved to dynamic port numbers.
ftpserver.pl is adjusted to load test instructions and test number from
the preprocessed test file.
FILEFORMAT.md now documents the new base64 encoding syntax.
Reported-by: Marcel Raad
Fixes#5761Closes#5775
Previously any connect-only connections in a multi handle would be kept
alive until the multi handle was closed. Since these connections cannot
be re-used, they can be marked for closure when the associated easy
handle is removed from the multi handle.
Closes#5749
include zstd curl patch for Makefile.m32 from vszakats
and include Add CMake support for zstd from Peter Wu
Helped-by: Viktor Szakats
Helped-by: Peter Wu
Closes#5453
Rely on tests asking the names to get refused instead - test servers
should be as dumb as possible. Edited test 914, 955 and 959 accordingly.
Closes#5639
Instead of discussing if there's value or meaning (implied or not) in
the colors, let's use words without the same possibly negative
associations.
Closes#5546
When the method is updated inside libcurl we must still not change the
method as set by the user as then repeated transfers with that same
handle might not execute the same operation anymore!
This fixes the libcurl part of #5462
Test 1633 added to verify.
Closes#5499
This test verifies the -w %json output and the test case includes a full
generated "blob". If there's no proxy support built into libcurl, it
will return an error for proxy related info variables and they will not
be included in the json, thus causing a mismatch and this test fails.
Reported-by: Marc Hörsken
Fixes#5501Closes#5502
... and free it as soon as the transfer is done. It removes the extra
alloc when a new size is set with setopt() and reduces memory for unused
easy handles.
In addition: the closure_handle now doesn't use an allocated buffer at
all but the smallest supported size as a stack based one.
Closes#5472
They're only limited to the maximum string input restrictions, not to
256 bytes.
Added test 1178 to verify
Reported-by: Will Roberts
Fixes#5448Closes#5449
curl would previously show "curl: Saved to filename 'name from header'"
if -J was used and a name was picked from the Content-Disposition
header. That output could interfer with other stdout output, such as -w.
This commit removes that output line.
Bug: https://curl.haxx.se/mail/archive-2020-05/0044.html
Reported-by: Коваленко Анатолий Викторович
Closes#5435
Added test 971 to verify that the list is in sync with the files in
cmdline-opts. The check also verifies that .d-files that uses Added:
specify the same version number as the options-in-versions file does.
Closes#5381
The tftpd server may still be busy if the total timeout of
25 seconds has not been reached or no sread error was received
during or after the execution of the timeout test 1238.
Once the next TFTP test comes around (eg. 1242 or 1243),
those will fail because the tftpd server is still waiting
on data from curl due to the UDP protocol being stateless
and having no connection close. On Linux this error may not
happen, because ICMP errors generated due to a swrite error
can also be returned async on the next sread call instead.
Therefore we will now just kill the tftpd server after test
1238 to make sure that the following tests are not affected.
This enables us to no longer ignore tests 1242, 1243, 2002
and 2003 on the CI platforms CirrusCI and AppVeyor.
Assisted-by: Peter Wu
Closes#5364
OpenSSH for Windows shows group and other/world permissions as *,
because those concepts do not exist on Windows. It also does not
show the current or parent directory, so we just ignore those.
Reviewed-by: Daniel Stenberg
Closes#5328
A common set of functions instead of many separate implementations for
creating buffers that can grow when appending data to them. Existing
functionality has been ported over.
In my early basic testing, the total number of allocations seem at
roughly the same amount as before, possibly a few less.
See docs/DYNBUF.md for a description of the API.
Closes#5300
OpenSSH for Windows requires paths in the format of /C:/
instead of the pseudo-POSIX paths /cygdrive/c/ or just /c/
Reviewed-by: Daniel Stenberg
Closes#5298
Several tests referred to specific server ports even when the test
didn't actually use that server or specify that it's needed. In such
cases, the test harness substitutes the text "[not running]" as the port
number which causes many such tests to fail due to the inability to
parse the URL. These tests are changed to use %NOLISTENPORT which will
always be substituted correctly.
More connection cache accesses are protected by locks.
CONNCACHE_* is a beter prefix for the connection cache lock macros.
Curl_attach_connnection: now called as soon as there's a connection
struct available and before the connection is added to the connection
cache.
Curl_disconnect: now assumes that the connection is already removed from
the connection cache.
Ref: #4915Closes#5009
Prior to this change if there was a 303 reply to a PUT request then
the subsequent request to respond to that redirect would also be a PUT.
It was determined that was most likely incorrect based on the language
of the RFCs. Basically 303 means "see other" resource, which implies it
is most likely not the same resource, therefore we should not try to PUT
to that different resource.
Refer to the discussions in #5237 and #5248 for more information.
Fixes https://github.com/curl/curl/issues/5237
Closes https://github.com/curl/curl/pull/5248
The purpose with this variable is to provide a port number that is
reasonably likely to not have a listener on the local host so that tests
can try connect failures against it. It uses port 47 - "reserved"
according to IANA.
Updated six tests to use it instead of the previous different ports.
Assisted-by: Emil Engler
Closes#5270
The runtests script now always performs variable replacement on the
entire test source file before the test gets executed, and saves the
updated version in a temporary file (log/test[num]) so that all test
case readers/servers can use that version (if present) and thus enjoy
the powers of test case variable substitution.
This is necessary to allow complete port number freedom.
Test 309 is updated to work with a non-fixed port number thanks to this.
- Ignore intermediate progress updates.
- Support locales that use a character other than period as decimal
separator (eg 100,0%).
test1148 checks that the progress finishes at 100% and has the right
bar width. Prior to this change the test assumed that the only progress
reported for such a quick transfer was 100%, however in rare instances
(like in the CI where transfer time can slow considerably) there may be
intermediate updates. For example, below is stderrlog1148 from a failed
CI run with explicit \r and \n added (it is one line; broken up so that
it's easier to understand).
\r
\r################################## 48.3%
\r######################################################################## 100.0%
\n
Closes https://github.com/curl/curl/pull/5194
test1294: "split request" being when the entire request isn't sent in
the first go, and the remainder is sent in the PERFORM state. A GET
request is otherwise not sending anything during PERFORM.
test1295: same kind of split but with POST
Closes#5197
This change inverse the order of processing for the --etag-compare and
--etag-save option to process first --etag-compare. This in turn allows
to use the same file name to compare and save an etag.
The original behavior of not failing if the etag file does not exists is
conserved.
Fixes#5179Closes#5180
Load long values correctly (e.g. for http_code).
Use curl_off_t (not long) for:
- size_download (CURLINFO_SIZE_DOWNLOAD_T)
- size_upload (CURLINFO_SIZE_UPLOAD_T)
The unit for these values is bytes/second, not microseconds:
- speed_download (CURLINFO_SPEED_DOWNLOAD_T)
- speed_upload (CURLINFO_SPEED_UPLOAD_T)
Fixes#5131Closes#5152
Reported by the new script 'scripts/copyright.pl'. The script has a
regex whitelist for the files that don't need copyright headers.
Removed three (mostly usesless) README files from docs/
Closes#5141
- send more data to make problems more obvious
- don't start the data with minus, it makes diffs harder to read
- skip the headers in the stdout comparison
- save to a file name to also verify 'filename_effective'
Ref: #5131
Update smbserver.py and negtelnetserver.py to be compatible with
Python 3 while staying backwards-compatible to support Python 2.
Fix string encoding and handling of echoed and transferred data.
Tested with both Python 2.7.17 and Python 3.7.7
Reported-by: Daniel Stenberg
Assisted-by: Kamil Dudka
Reviewed-by: Marcel Raad
Fixes#5104Closes#5110
- Implement new option CURLSSLOPT_REVOKE_BEST_EFFORT and
--ssl-revoke-best-effort to allow a "best effort" revocation check.
A best effort revocation check ignores errors that the revocation check
was unable to take place. The reasoning is described in detail below and
discussed further in the PR.
---
When running e.g. with Fiddler, the schannel backend fails with an
unhelpful error message:
Unknown error (0x80092012) - The revocation function was unable
to check revocation for the certificate.
Sadly, many enterprise users who are stuck behind MITM proxies suffer
the very same problem.
This has been discussed in plenty of issues:
https://github.com/curl/curl/issues/3727,
https://github.com/curl/curl/issues/264, for example.
In the latter, a Microsoft Edge developer even made the case that the
common behavior is to ignore issues when a certificate has no recorded
distribution point for revocation lists, or when the server is offline.
This is also known as "best effort" strategy and addresses the Fiddler
issue.
Unfortunately, this strategy was not chosen as the default for schannel
(and is therefore a backend-specific behavior: OpenSSL seems to happily
ignore the offline servers and missing distribution points).
To maintain backward-compatibility, we therefore add a new flag
(`CURLSSLOPT_REVOKE_BEST_EFFORT`) and a new option
(`--ssl-revoke-best-effort`) to select the new behavior.
Due to the many related issues Git for Windows and GitHub Desktop, the
plan is to make this behavior the default in these software packages.
The test 2070 was added to verify this behavior, adapted from 310.
Based-on-work-by: georgeok <giorgos.n.oikonomou@gmail.com>
Co-authored-by: Markus Olsson <j.markus.olsson@gmail.com>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Closes https://github.com/curl/curl/pull/4981
Makes curl_easy_getinfo() of "variable" numerical content instead return
the number set in the env variable `CURL_TIME`.
Makes curl_version() of "variable" textual content. This guarantees a
stable version string which can be tested against. Environment variable
`CURL_VERSION` defines the content.
Assisted-by: Mathias Gumz
The test uses SRP to "a server not supporting it" but modern stunnel
versions will silently accept it and remain happy. The test is therefore
faulty.
I haven't figured out how to make stunnel explicitly reject SRP-using
connects.
Reported-by: Marc Hörsken
Fixes#5105Closes#5113
Users of the SMB tests will have to install impacket manually.
Reasoning: our in-tree version of impacket was quite outdated
and only compatible with Python 2 which is already end-of-life.
Upgrading to Python 3 and a compatible impacket version would
require to import additional Python-only and CPython-extension
dependencies. This would have hindered portability enormously.
Closes#5094
When extracting a <section> <part> and there's no </part> before
</section>, this now outputs an error and returns a wrong string to
make users spot the mistake.
Ref: #5070Closes#5071
New test 666 checks this is effective.
As upload buffer size is significant in this kind of tests, shorten it
in similar test 652.
Fixes#4860Closes#4833
Reported-by: RuurdBeerstra on github
Input buffer filling may delay the data sending if data reads are slow.
To overcome this problem, file and callback data reads do not accumulate
in buffer anymore. All other data (memory data and mime framing) are
considered as fast and still concatenated in buffer.
As this may highly impact performance in terms of data overhead, an early
end of part data check is added to spare a read call.
When encoding a part's data, an encoder may require more bytes than made
available by a single read. In this case, the above rule does not apply
and reads are performed until the encoder is able to deliver some data.
Tests 643, 644, 645, 650 and 654 have been adapted to the output data
changes, with test data size reduced to avoid the boredom of long lists of
1-byte chunks in verification data.
New test 667 checks mimepost using single-byte read callback with encoder.
New test 668 checks the end of part data early detection.
Fixes#4826
Reported-by: MrdUkk on github
In case a read callback returns a status (pause, abort, eof,
error) instead of a byte count, drain the bytes read so far but
remember this status for further processing.
Takes care of not losing data when pausing, and properly resume a
paused mime structure when requested.
New tests 670-673 check unpausing cases, with easy or multi
interface and mime or form api.
Fixes#4813
Reported-by: MrdUkk on github
