- enable in the build (configure)
- header parsing
- host name lookup
- unit tests for the above
- CI build
- CURL_VERSION_HSTS bit
- curl_version_info support
- curl -V output
- curl-config --features
- CURLOPT_HSTS_CTRL
- man page for CURLOPT_HSTS_CTRL
- curl --hsts (sets CURLOPT_HSTS_CTRL and works with --libcurl)
- man page for --hsts
- save cache to disk
- load cache from disk
- CURLOPT_HSTS
- man page for CURLOPT_HSTS
- added docs/HSTS.md
- fixed --version docs
- adjusted curl_easy_duphandle
Closes#5896
- Add regex that strips http/2 server header name to those http/2 tests
that don't already have it.
- Improve that regex in all http/2 tests.
Tests 358 and 359 were failing for me before this change on a system
that uses an older version of nghttpx which includes its version number
in the server header.
Closes https://github.com/curl/curl/pull/6139
... to make sure such tests are run with valgrind. Suppress the zstd
valgrind warnings we get with version 1.3.3 on Ubuntu 18.04 (for debug
and non-debug builds).
Closes#6105
... and remove the objnames scripts they tested. They're not used for
anything anymore so testing them serves no purpose!
Reported-by: Marc Hörsken
Fixes#6080Closes#6081
Update appveyor.yml to set env variable TFLAGS and run tests
Remove curly braces due to CMake error (${TFLAGS} -> $TFLAGS)
Move testdeps build to build step (per review comments)
Reviewed-by: Marc Hörsken
Closes#6066Fixes#6052
Avoid using our own, potentially installed, curl for
the test reporting APIs in case it is broken.
Reviewed-by: Daniel Stenberg
Preparation for #6049Closes#6063
... and use this new keywords in all the test files larger than 50K to reduce
their sizes and make them a lot easier to read and understand.
Closes#6040
Added test 348 to verify. Added a 'STOR' command to the test FTP
server to enable test 348. Documented the command in FILEFORMAT.md
Reported-by: Duncan Wilcox
Fixes#6016Closes#6017
When using HTTPS proxy, SSL is used but not in the view of the FTP
protocol handler itself so separate the connection's use of SSL from the
FTP control connection's sue.
Reported-by: Mingtao Yang
Fixes#5523Closes#6006
This is primarily interesting for cases where CURLOPT_NOBODY is set as
previously curl would not return an error for this case.
MDTM getting 550 now also returns this error (it returned
CURLE_FTP_COULDNT_RETR_FILE before) in order to unify return codes for
missing files across protocols and specific FTP commands.
libcurl already returns error on a 550 as a MDTM response (when
CURLOPT_FILETIME is set). If CURLOPT_NOBODY is not set, an error would
happen subsequently anyway since the RETR command would fail.
Add test 1913 and 1914 to verify. Updated several tests accordingly due
to the updated SIZE behavior.
Reported-by: Tomas Berger
Fixes#5953Closes#5957
configure --enable-debug now enables -Wassign-enum with clang,
identifying several enum "abuses" also fixed.
Reported-by: Gisle Vanem
Bug: 879007f811 (commitcomment-42087553)Closes#5929
To ease identification of tests the test number is added to
the test case title in order to have it on the Azure DevOps
Analytics pages and reports which currently do not show it.
Bump test case revision to make Azure DevOps update titles.
Closes#5927
The cache content is not duplicated, like other caches, but the setting
and specified file name are.
Test 1908 is extended to verify this somewhat. Since the duplicated
handle gets the same file name, the test unfortunately overwrites the
same file twice (with different contents) which makes it hard to check
automatically.
Closes#5923
A shared connection cache is not thread-safe is a known issue. Stop
testing this until we believe this issue is addressed. Reduces
occasional test failures we don't care about.
The test code in lib1541.c is left in git to allow us to restore it when
we get to fix this.
Closes#5922
Remove the tests 530, 584, 1900, 1901, 1902, 1903 and 2033. They were
previously disabled.
The Pipelining code was removed from curl in commit 2f44e94efb,
April 2019.
Closes#5921
IPv6, telnet and now also the multi API require WinSock
version 2 which is available starting with Windows 95.
Therefore we think it is time to drop support for version 1.
Reviewed-by: Marcel Raad
Reviewed-by: Jay Satiro
Reviewed-by: Daniel Stenberg
Reviewed-by: Viktor Szakats
Follow up to #5634Closes#5854
Since we already use the base64 package since a while back, we can just
as well switch to that here too.
It also happens to use the exact same function name, which otherwise
causes a run-time warning.
Reported-by: Marc Hörsken
Fixes#5885Closes#5887
Learn from the way Cygwin handles and maps the WinSock events
to simulate correct and complete poll and select behaviour
according to Richard W. Stevens Network Programming book.
Follow up to #5867Closes#5879
const struct curl_easyoption *curl_easy_option_by_name(const char *name);
const struct curl_easyoption *curl_easy_option_by_id (CURLoption id);
const struct curl_easyoption *
curl_easy_option_next(const struct curl_easyoption *prev);
The purpose is to provide detailed enough information to allow for
example libcurl bindings to get option information at run-time about
what easy options that exist and what arguments they expect.
Assisted-by: Jeroen Ooms
Closes#5365
Currently the file tests/libtest/lib3010 is not getting
ignored by git. This fixes it by adding the 3XXX area to
the according .gitignore file.
Closes#5859
Failures clearly returned from a (SOCKS) proxy now causes this return
code. Previously the situation was not very clear as what would be
returned and when.
In addition: when this error code is returned, an application can use
CURLINFO_PROXY_ERROR to query libcurl for the detailed error, which then
returns a value from the new 'CURLproxycode' enum.
Closes#5770
copy the LDFLAGS approach for adding same option with `libhostname` in
`libtest/Makefile.am`:
- init `libstubgss_la_LDFLAGS_EXTRA` variable,
- add option to variable inside conditional,
- use variable in `libstubgss_la_LDFLAGS`
Fixes#5819Closes#5820
The CMake option is now called CMAKE_USE_SCHANNEL
The winbuild flag is USE_SCHANNEL
The CI jobs and build scripts only use the new names and the new name
options
Tests now require 'Schannel' (when necessary)
Closes#5795
This allows us to make test cases to use base64 at run-time and still
use and verify information determined at run-time, such as the IMAP test
server's port number in test 842.
This change makes 12 tests run again that basically never ran since we
moved to dynamic port numbers.
ftpserver.pl is adjusted to load test instructions and test number from
the preprocessed test file.
FILEFORMAT.md now documents the new base64 encoding syntax.
Reported-by: Marcel Raad
Fixes#5761Closes#5775
Previously any connect-only connections in a multi handle would be kept
alive until the multi handle was closed. Since these connections cannot
be re-used, they can be marked for closure when the associated easy
handle is removed from the multi handle.
Closes#5749
Without the -D command line option, checksrc.pl won't know which
directory to load the ".checksrc" file from when building out of the
source tree.
Reported-by: Marcel Raad
Fixes#5715Closes#5755
They're not thread-safe so they should not be used in libcurl code.
Explictly enabled when deemed necessary and in examples and tests
Reviewed-by: Nicolas Sterchele
Closes#5732
Stack overflows can occur with precisions for integers and floats.
Proof of concepts:
- curl_mprintf("%d, %.*1$d", 500, 1);
- curl_mprintf("%d, %+0500.*1$f", 500, 1);
Ideally, compile with -fsanitize=address which makes this undefined
behavior a bit more defined for debug purposes.
The format strings are valid. The overflows occur due to invalid
arguments. If these arguments are variables with contents controlled
by an attacker, the function's stack can be corrupted.
Also see CVE-2016-9586 which partially fixed the float aspect.
Signed-off-by: Tobias Stoeckmann <tobias@stoeckmann.org>
Closes https://github.com/curl/curl/pull/5722
timeval::tv_usec might be a 32-bit integer and timespec::tv_nsec might
be a 64-bit integer. This is the case when building for recent macOS
versions, for example. Just treat tv_usec as an int, which should
hopefully always be sufficient on systems with
`HAVE_CLOCK_GETTIME_MONOTONIC`.
Closes https://github.com/curl/curl/pull/5695
include zstd curl patch for Makefile.m32 from vszakats
and include Add CMake support for zstd from Peter Wu
Helped-by: Viktor Szakats
Helped-by: Peter Wu
Closes#5453
Rely on tests asking the names to get refused instead - test servers
should be as dumb as possible. Edited test 914, 955 and 959 accordingly.
Closes#5639
Updated terminology in docs, comments and phrases to refer to C strings
as "null-terminated". Done to unify with how most other C oriented docs
refer of them and what users in general seem to prefer (based on a
single highly unscientific poll on twitter).
Reported-by: coinhubs on github
Fixes#5598Closes#5608
Instead of discussing if there's value or meaning (implied or not) in
the colors, let's use words without the same possibly negative
associations.
Closes#5546
To reduce the amount of allocations needed for creating a Curl_addrinfo
struct, make a single larger malloc instead of three separate smaller
ones.
Closes#5533
When the method is updated inside libcurl we must still not change the
method as set by the user as then repeated transfers with that same
handle might not execute the same operation anymore!
This fixes the libcurl part of #5462
Test 1633 added to verify.
Closes#5499
This test verifies the -w %json output and the test case includes a full
generated "blob". If there's no proxy support built into libcurl, it
will return an error for proxy related info variables and they will not
be included in the json, thus causing a mismatch and this test fails.
Reported-by: Marc Hörsken
Fixes#5501Closes#5502
... and free it as soon as the transfer is done. It removes the extra
alloc when a new size is set with setopt() and reduces memory for unused
easy handles.
In addition: the closure_handle now doesn't use an allocated buffer at
all but the smallest supported size as a stack based one.
Closes#5472
GCC 10 warns about this with warning: implicit conversion
from 'SANITIZEcode' to 'CURLcode' [-Wenum-conversion]
Since 'expected_result' is not really of type 'CURLcode' and
it is not exposed in any way, we can just use 'SANITIZEcode'.
Reviewed-by: Daniel Stenberg
Reviewed-by: Marcel Raad
Closes#5476
Since curl_setup.h now makes use of curlx_win32_fopen for Win32
builds with USE_WIN32_LARGE_FILES or USE_WIN32_SMALL_FILES defined,
we need to include the relevant files for tests using fopen,
because the libtest sources are also including curl_setup.h
Reviewed-by: Marcel Raad
Reviewed-by: Daniel Stenberg
Follow up to #3784 (ffdddb45d9)
Closes#5475
They're only limited to the maximum string input restrictions, not to
256 bytes.
Added test 1178 to verify
Reported-by: Will Roberts
Fixes#5448Closes#5449
Fixed the alt-svc parser to treat a newline as end of line.
The unit tests in test 1654 were done without CRLF and thus didn't quite
match the real world. Now they use CRLF as well.
Reported-by: Peter Wu
Assisted-by: Peter Wu
Assisted-by: Jay Satiro
Fixes#5445Closes#5446
curl would previously show "curl: Saved to filename 'name from header'"
if -J was used and a name was picked from the Content-Disposition
header. That output could interfer with other stdout output, such as -w.
This commit removes that output line.
Bug: https://curl.haxx.se/mail/archive-2020-05/0044.html
Reported-by: Коваленко Анатолий Викторович
Closes#5435
- Stick to a single unified way to use structs
- Make checksrc complain on 'typedef struct {'
- Allow them in tests, public headers and examples
- Let MD4_CTX, MD5_CTX, and SHA256_CTX typedefs remain as they actually
typedef different types/structs depending on build conditions.
Closes#5338
If it can't be found in the request. Also support --cmdfile to set it to
a custom file name.
runtests.pl always writes this file with the test number in it since a
while back.
Added test 971 to verify that the list is in sync with the files in
cmdline-opts. The check also verifies that .d-files that uses Added:
specify the same version number as the options-in-versions file does.
Closes#5381
* runtests.pl:
- Fix out-of-tree build under CMake when srcdir is not set. Default
srcdir to the location of runtests.pl.
- Add a hack to allow CMake to use the TFLAGS option as documented
in tests/README and used in scripts/travis/script.sh.
* Bump CMake version to 3.2 for USES_TERMINAL, dropping Debian Jessie
support (no one should care, it is already EOL.).
* Remove CTest since it defines its own 'test' target with no tests
since all unittests are already broken and not built by default.
* Add new test targets based on the options from Makefile.am. Since
new test targets are rarely added, I opted for duplicating the
runtests.pl options as opposed to creating a new Makefile.inc file.
Use top-level target names (test-x) instead of x-test since that is
used by CI and others.
Closes#5358
The default target should only build libcurl and curl. Add a dedicated
'testdeps' target which will be used later when running tests. Note that
unittests are currently broken in CMake and already excluded.
Closes#5368
The tftpd server may still be busy if the total timeout of
25 seconds has not been reached or no sread error was received
during or after the execution of the timeout test 1238.
Once the next TFTP test comes around (eg. 1242 or 1243),
those will fail because the tftpd server is still waiting
on data from curl due to the UDP protocol being stateless
and having no connection close. On Linux this error may not
happen, because ICMP errors generated due to a swrite error
can also be returned async on the next sread call instead.
Therefore we will now just kill the tftpd server after test
1238 to make sure that the following tests are not affected.
This enables us to no longer ignore tests 1242, 1243, 2002
and 2003 on the CI platforms CirrusCI and AppVeyor.
Assisted-by: Peter Wu
Closes#5364
setjmp.h should only be included if HAVE_SETJMP_H is defined.
Add additional log statements to see wether reads and writes
are blocking or finishing before an alarm signal is received.
Assisted-by: Peter Wu
Part of #5364
OpenSSH for Windows shows group and other/world permissions as *,
because those concepts do not exist on Windows. It also does not
show the current or parent directory, so we just ignore those.
Reviewed-by: Daniel Stenberg
Closes#5328
A common set of functions instead of many separate implementations for
creating buffers that can grow when appending data to them. Existing
functionality has been ported over.
In my early basic testing, the total number of allocations seem at
roughly the same amount as before, possibly a few less.
See docs/DYNBUF.md for a description of the API.
Closes#5300
Remove many one second sleeps that were done *after* each newly started
test server already has been verified. They should not have any purpose
there.
Closes#5323
This makes sure that select_ws behaves similar to real select
which stops waiting on a signal handler being triggered.
This makes it possible to gracefully stop sockfilt.exe on
Windows with taskkill /IM sockfilt.exe (without /F force flag).
Reviewed-by: Jay Satiro
Part of #5260
This commit adds a global exit event to the test servers that
Windows-specific wait routines can use to get triggered if the
program was signaled to be terminated, eg. select_ws in sockfilt.c
The exit event will be managed by the signal handling code and is
set to not reset automatically to support multiple wait routines.
Reviewed-by: Jay Satiro
Closes#5260
Use raise to trigger signal handler instead of calling it
directly and causing potential unexpected control flow.
Reviewed-by: Jay Satiro
Part of #5260
OpenSSH for Windows requires paths in the format of /C:/
instead of the pseudo-POSIX paths /cygdrive/c/ or just /c/
Reviewed-by: Daniel Stenberg
Closes#5298
Several tests referred to specific server ports even when the test
didn't actually use that server or specify that it's needed. In such
cases, the test harness substitutes the text "[not running]" as the port
number which causes many such tests to fail due to the inability to
parse the URL. These tests are changed to use %NOLISTENPORT which will
always be substituted correctly.
More connection cache accesses are protected by locks.
CONNCACHE_* is a beter prefix for the connection cache lock macros.
Curl_attach_connnection: now called as soon as there's a connection
struct available and before the connection is added to the connection
cache.
Curl_disconnect: now assumes that the connection is already removed from
the connection cache.
Ref: #4915Closes#5009
As stunnel is an external tool and it has no specific option to export
the actually used port number when asked to listen to 0, runtests
instead iterates over ten randomly picked high number ports and sticks
to the first one stunnel can listen to.
Closes#5267
These were commented out in e9dd099870 when Makefile.inc was included
instead. 11 years have passed since then and the commented code is of
course very outdated. Remove it to avoid confusion.
Closes#5311
Since the http2 test server is a mere proxy that needs to know about the
dynamic port the HTTP server is using, it too needs to get restarted
when the http server is killed.
A regression caused by 80d6515.
Fixes#5289Closes#5291
Prior to this change if there was a 303 reply to a PUT request then
the subsequent request to respond to that redirect would also be a PUT.
It was determined that was most likely incorrect based on the language
of the RFCs. Basically 303 means "see other" resource, which implies it
is most likely not the same resource, therefore we should not try to PUT
to that different resource.
Refer to the discussions in #5237 and #5248 for more information.
Fixes https://github.com/curl/curl/issues/5237
Closes https://github.com/curl/curl/pull/5248
The purpose with this variable is to provide a port number that is
reasonably likely to not have a listener on the local host so that tests
can try connect failures against it. It uses port 47 - "reserved"
according to IANA.
Updated six tests to use it instead of the previous different ports.
Assisted-by: Emil Engler
Closes#5270
This fixes the error: 'for' loop initial declaration used outside C99
mode by declaring the loop increment variable in the beginning of the
block instead of inside the for loop.
Fixes#5254
Reviewed-by: Daniel Gustafsson <daniel@yesql.se>
FTP, IMAP, POP3, SMTP and their IPv6 versions are now all on dynamic
ports
Test 842-845 are unfortunately a bit hard to move over to this concept
right now and require "default port" still...
The runtests script now always performs variable replacement on the
entire test source file before the test gets executed, and saves the
updated version in a temporary file (log/test[num]) so that all test
case readers/servers can use that version (if present) and thus enjoy
the powers of test case variable substitution.
This is necessary to allow complete port number freedom.
Test 309 is updated to work with a non-fixed port number thanks to this.
Since the Windows versions of both programs would write Windows
PIDs to their pidfiles which we cannot handle, we need to use
our known perl.exe Cygwin/msys PID together with exec() in order
to tie the spawned processes to the existance of our perl.exe
The perl.exe that is executing secureserver.pl and sshserver.pl
has a Cygwin/msys PID, because it is started inside Cygwin/msys.
Related to #5188
- Ignore intermediate progress updates.
- Support locales that use a character other than period as decimal
separator (eg 100,0%).
test1148 checks that the progress finishes at 100% and has the right
bar width. Prior to this change the test assumed that the only progress
reported for such a quick transfer was 100%, however in rare instances
(like in the CI where transfer time can slow considerably) there may be
intermediate updates. For example, below is stderrlog1148 from a failed
CI run with explicit \r and \n added (it is one line; broken up so that
it's easier to understand).
\r
\r################################## 48.3%
\r######################################################################## 100.0%
\n
Closes https://github.com/curl/curl/pull/5194
With this bit set, my mac successfully resolves "ip6-localhost" when in
fact there is no such host known to my machine! That in turn made test
241 wrongly execute and fail.
Closes#5202
test1294: "split request" being when the entire request isn't sent in
the first go, and the remainder is sent in the PERFORM state. A GET
request is otherwise not sending anything during PERFORM.
test1295: same kind of split but with POST
Closes#5197
This change inverse the order of processing for the --etag-compare and
--etag-save option to process first --etag-compare. This in turn allows
to use the same file name to compare and save an etag.
The original behavior of not failing if the etag file does not exists is
conserved.
Fixes#5179Closes#5180
Replace the incomplete workaround regarding FD_CLOSE
only signalling once by instead doing a pre-check with
standard select and storing the result for later use.
select keeps triggering on closed sockets on Windows while
WSAEventSelect fires only once with data still available.
By doing the pre-check we do not run in a deadlock
due to waiting forever for another FD_CLOSE event.
Fix race-condition of waiting threads finishing while events are
already being processed which lead to invalid or skipped events.
Use mutex to check for one event at a time or do post-processing.
In addition to mutex-based locking use specific event as signal.
Closes#5156
Load long values correctly (e.g. for http_code).
Use curl_off_t (not long) for:
- size_download (CURLINFO_SIZE_DOWNLOAD_T)
- size_upload (CURLINFO_SIZE_UPLOAD_T)
The unit for these values is bytes/second, not microseconds:
- speed_download (CURLINFO_SPEED_DOWNLOAD_T)
- speed_upload (CURLINFO_SPEED_UPLOAD_T)
Fixes#5131Closes#5152
Reported by the new script 'scripts/copyright.pl'. The script has a
regex whitelist for the files that don't need copyright headers.
Removed three (mostly usesless) README files from docs/
Closes#5141
- send more data to make problems more obvious
- don't start the data with minus, it makes diffs harder to read
- skip the headers in the stdout comparison
- save to a file name to also verify 'filename_effective'
Ref: #5131
Update smbserver.py and negtelnetserver.py to be compatible with
Python 3 while staying backwards-compatible to support Python 2.
Fix string encoding and handling of echoed and transferred data.
Tested with both Python 2.7.17 and Python 3.7.7
Reported-by: Daniel Stenberg
Assisted-by: Kamil Dudka
Reviewed-by: Marcel Raad
Fixes#5104Closes#5110
- Implement new option CURLSSLOPT_REVOKE_BEST_EFFORT and
--ssl-revoke-best-effort to allow a "best effort" revocation check.
A best effort revocation check ignores errors that the revocation check
was unable to take place. The reasoning is described in detail below and
discussed further in the PR.
---
When running e.g. with Fiddler, the schannel backend fails with an
unhelpful error message:
Unknown error (0x80092012) - The revocation function was unable
to check revocation for the certificate.
Sadly, many enterprise users who are stuck behind MITM proxies suffer
the very same problem.
This has been discussed in plenty of issues:
https://github.com/curl/curl/issues/3727,
https://github.com/curl/curl/issues/264, for example.
In the latter, a Microsoft Edge developer even made the case that the
common behavior is to ignore issues when a certificate has no recorded
distribution point for revocation lists, or when the server is offline.
This is also known as "best effort" strategy and addresses the Fiddler
issue.
Unfortunately, this strategy was not chosen as the default for schannel
(and is therefore a backend-specific behavior: OpenSSL seems to happily
ignore the offline servers and missing distribution points).
To maintain backward-compatibility, we therefore add a new flag
(`CURLSSLOPT_REVOKE_BEST_EFFORT`) and a new option
(`--ssl-revoke-best-effort`) to select the new behavior.
Due to the many related issues Git for Windows and GitHub Desktop, the
plan is to make this behavior the default in these software packages.
The test 2070 was added to verify this behavior, adapted from 310.
Based-on-work-by: georgeok <giorgos.n.oikonomou@gmail.com>
Co-authored-by: Markus Olsson <j.markus.olsson@gmail.com>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Closes https://github.com/curl/curl/pull/4981
Makes curl_easy_getinfo() of "variable" numerical content instead return
the number set in the env variable `CURL_TIME`.
Makes curl_version() of "variable" textual content. This guarantees a
stable version string which can be tested against. Environment variable
`CURL_VERSION` defines the content.
Assisted-by: Mathias Gumz
The test uses SRP to "a server not supporting it" but modern stunnel
versions will silently accept it and remain happy. The test is therefore
faulty.
I haven't figured out how to make stunnel explicitly reject SRP-using
connects.
Reported-by: Marc Hörsken
Fixes#5105Closes#5113
Users of the SMB tests will have to install impacket manually.
Reasoning: our in-tree version of impacket was quite outdated
and only compatible with Python 2 which is already end-of-life.
Upgrading to Python 3 and a compatible impacket version would
require to import additional Python-only and CPython-extension
dependencies. This would have hindered portability enormously.
Closes#5094
