The previous sleep for retries would block all other concurrent
transfers. Starting now, the retry will instead be properly marked to
not get restarted until after the delay time but other transfers can
still continue in the mean time.
Closes#5917
They're not thread-safe so they should not be used in libcurl code.
Explictly enabled when deemed necessary and in examples and tests
Reviewed-by: Nicolas Sterchele
Closes#5732
Previously a file that isn't user-readable but is user-writable would
not be properly avoided and would get overwritten.
Reported-by: BrumBrum on hackerone
Assisted-by: Jay Satiro
Bug: https://hackerone.com/reports/926638Closes#5731
timeval::tv_usec might be a 32-bit integer and timespec::tv_nsec might
be a 64-bit integer. This is the case when building for recent macOS
versions, for example. Just treat tv_usec as an int, which should
hopefully always be sufficient on systems with
`HAVE_CLOCK_GETTIME_MONOTONIC`.
Closes https://github.com/curl/curl/pull/5695
It confuses code analyzers with its use of -1 for unsigned value. Also,
a check that's not normally used in strdup() code - and not necessary.
Closes#5697
... to use the maximum value for 'size_t' when detecting integer overflow.
Changed the limit to max/4 as already that seems unreasonably large.
Codacy didn't like the previous approach.
Closes#5683
include zstd curl patch for Makefile.m32 from vszakats
and include Add CMake support for zstd from Peter Wu
Helped-by: Viktor Szakats
Helped-by: Peter Wu
Closes#5453
Updated terminology in docs, comments and phrases to refer to C strings
as "null-terminated". Done to unify with how most other C oriented docs
refer of them and what users in general seem to prefer (based on a
single highly unscientific poll on twitter).
Reported-by: coinhubs on github
Fixes#5598Closes#5608
Background:
148534d added CURLSSLOPT_NATIVE_CA to use the Windows OS certificate
store in libcurl w/ OpenSSL on Windows. CURLSSLOPT_NATIVE_CA overrides
CURLOPT_CAINFO if both are set. The curl tool will fall back to
CURLSSLOPT_NATIVE_CA if it could not find a certificate bundle to set
via CURLOPT_CAINFO.
Problem:
libcurl may be built with hardcoded paths to a certificate bundle or
directory, and if CURLSSLOPT_NATIVE_CA is used then those paths are
ignored.
Solution:
A solution is still being discussed but since there's an impending
release this commit removes using CURLSSLOPT_NATIVE_CA in the curl tool.
Ref: https://github.com/curl/curl/issues/5585
... and not as a "glob". Now done by passing the supposed host to the
URL parser which supposedly will do a better job at identifying "real"
numerical IPv6 addresses.
Reported-by: puckipedia on github
Fixes#5576Closes#5579
Prior to this change in Windows Unicode builds most parsed options would
not be freed.
Found using _CrtDumpMemoryLeaks().
Ref: https://github.com/curl/curl/issues/5545
curl would previously show "curl: Saved to filename 'name from header'"
if -J was used and a name was picked from the Content-Disposition
header. That output could interfer with other stdout output, such as -w.
This commit removes that output line.
Bug: https://curl.haxx.se/mail/archive-2020-05/0044.html
Reported-by: Коваленко Анатолий Викторович
Closes#5435
This change introduces a generic way to provide binary data in setopt
options, called BLOBs.
This change introduces these new setopts:
CURLOPT_ISSUERCERT_BLOB, CURLOPT_PROXY_SSLCERT_BLOB,
CURLOPT_PROXY_SSLKEY_BLOB, CURLOPT_SSLCERT_BLOB and CURLOPT_SSLKEY_BLOB.
Reviewed-by: Daniel Stenberg
Closes#5357
- Stick to a single unified way to use structs
- Make checksrc complain on 'typedef struct {'
- Allow them in tests, public headers and examples
- Let MD4_CTX, MD5_CTX, and SHA256_CTX typedefs remain as they actually
typedef different types/structs depending on build conditions.
Closes#5338
- use `wmain` instead of `main` when `_UNICODE` is defined [0]
- define `argv_item_t` as `wchar_t *` in this case
- use the curl_multibyte gear to convert the command-line arguments to
UTF-8
This makes it possible to pass parameters with characters outside of
the current locale on Windows, which is required for some tests, e.g.
the IDN tests. Out of the box, this currently only works with the
Visual Studio project files, which default to Unicode, and winbuild
with the `ENABLE_UNICODE` option.
[0] https://devblogs.microsoft.com/oldnewthing/?p=40643
Ref: https://github.com/curl/curl/issues/3747
Closes https://github.com/curl/curl/pull/3784
To avoid collision risks with private libcurl symbols when linked with
static versions (or just versions not hiding internal symbols).
Reported-by: hydra3333 on github
Fixes#5219Closes#5234
This change inverse the order of processing for the --etag-compare and
--etag-save option to process first --etag-compare. This in turn allows
to use the same file name to compare and save an etag.
The original behavior of not failing if the etag file does not exists is
conserved.
Fixes#5179Closes#5180
Load long values correctly (e.g. for http_code).
Use curl_off_t (not long) for:
- size_download (CURLINFO_SIZE_DOWNLOAD_T)
- size_upload (CURLINFO_SIZE_UPLOAD_T)
The unit for these values is bytes/second, not microseconds:
- speed_download (CURLINFO_SPEED_DOWNLOAD_T)
- speed_upload (CURLINFO_SPEED_UPLOAD_T)
Fixes#5131Closes#5152
Previously it rendered the page from files matching "*.d" in the correct
directory, which worked fine in git builds when the files were added but
made it easy to forget adding the files to the dist.
Now, only man page sections listed in DPAGES in Makefile.inc will be
used, thus "forcing" us to update this to get the man page right and get
it included in the dist at the same time.
Ref: #5146Closes#5149
Reported by the new script 'scripts/copyright.pl'. The script has a
regex whitelist for the files that don't need copyright headers.
Removed three (mostly usesless) README files from docs/
Closes#5141
Ensures that -K/--config inserts new items at the end of the list
instead of overwriting the second item, and that after a -K/--config
option has been parsed, the option parser's view of the current config
is update.
Fixes#5120Closes#5123
- Implement new option CURLSSLOPT_REVOKE_BEST_EFFORT and
--ssl-revoke-best-effort to allow a "best effort" revocation check.
A best effort revocation check ignores errors that the revocation check
was unable to take place. The reasoning is described in detail below and
discussed further in the PR.
---
When running e.g. with Fiddler, the schannel backend fails with an
unhelpful error message:
Unknown error (0x80092012) - The revocation function was unable
to check revocation for the certificate.
Sadly, many enterprise users who are stuck behind MITM proxies suffer
the very same problem.
This has been discussed in plenty of issues:
https://github.com/curl/curl/issues/3727,
https://github.com/curl/curl/issues/264, for example.
In the latter, a Microsoft Edge developer even made the case that the
common behavior is to ignore issues when a certificate has no recorded
distribution point for revocation lists, or when the server is offline.
This is also known as "best effort" strategy and addresses the Fiddler
issue.
Unfortunately, this strategy was not chosen as the default for schannel
(and is therefore a backend-specific behavior: OpenSSL seems to happily
ignore the offline servers and missing distribution points).
To maintain backward-compatibility, we therefore add a new flag
(`CURLSSLOPT_REVOKE_BEST_EFFORT`) and a new option
(`--ssl-revoke-best-effort`) to select the new behavior.
Due to the many related issues Git for Windows and GitHub Desktop, the
plan is to make this behavior the default in these software packages.
The test 2070 was added to verify this behavior, adapted from 310.
Based-on-work-by: georgeok <giorgos.n.oikonomou@gmail.com>
Co-authored-by: Markus Olsson <j.markus.olsson@gmail.com>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Closes https://github.com/curl/curl/pull/4981
This commit adds support to generate JSON via the writeout feature:
-w "%{json}"
It leverages the existing infrastructure as much as possible. Thus,
generating the JSON on STDERR is possible by:
-w "%{stderr}%{json}"
This implements a variant of
https://github.com/curl/curl/wiki/JSON#--write-out-json.
Closes#4870
Trying to return early from the function if no new transfers were added
would break the "morep" argument and cause issues. This could lead to
zero content "transfers" (within quotes since they would never be
started) when parallel-max was reduced.
Reported-by: Gavin Wong
Analyzed-by: Jay Satiro
Fixes#4937Closes#5112
Whilst lib\md4.c used this pre-processor, lib\md5.c and
src\tool_metalink.c did not and simply relied on the WIN32
pre-processor directive.
Reviewed-by: Marcel Raad
Closes#4955
- Change tool_util.c tvnow() for Windows to match more closely to
timeval.c Curl_now().
- Create a win32 init function for the tool, since some initialization
is required for the tvnow() changes.
Prior to this change the monotonic time function used by curl in Windows
was determined at build-time and not runtime. That was a problem because
when curl was built targeted for compatibility with old versions of
Windows (eg _WIN32_WINNT < 0x0600) it would use GetTickCount which wraps
every 49.7 days that Windows has been running.
This change makes curl behave similar to libcurl's tvnow function, which
determines at runtime whether the OS is Vista+ and if so calls
QueryPerformanceCounter instead. (Note QueryPerformanceCounter is used
because it has higher resolution than the more obvious candidate
GetTickCount64). The changes to tvnow are basically a copy and paste but
the types in some cases are different.
Ref: https://github.com/curl/curl/issues/3309
Closes https://github.com/curl/curl/pull/4847
- Deduplicate GetEnv() code.
- On Windows change ultimate call to use Windows API
GetEnvironmentVariable() instead of C runtime getenv().
Prior to this change both libcurl and the tool had their own GetEnv
which over time diverged. Now the tool's GetEnv is a wrapper around
curl_getenv (libcurl API function which is itself a wrapper around
libcurl's GetEnv).
Furthermore this change fixes a bug in that Windows API
GetEnvironmentVariable() is called instead of C runtime getenv() to get
the environment variable since some changes aren't always visible to the
latter.
Reported-by: Christoph M. Becker
Fixes https://github.com/curl/curl/issues/4774
Closes https://github.com/curl/curl/pull/4863
Make the tool check for alt-svc support at run-time and return error
accordingly if not present when the option is used.
Reported-by: Harry Sintonen
Closes#4878
The fixed-point math made us lose precision and thus a too high index
value could be used for outputting the hashtags which could overwrite
the newline.
The fix increases the precision in the sine table (*100) and the
associated position math.
Reported-by: Andrew Potter
Fixes#4849Closes#4850
As it was just unnecessary duplicated information already stored in the
'per_transfer' struct and that's around mostly anyway.
The duplicated pointer caused problems when the code flow was aborted
before the dupe was filled in and could cause a NULL pointer access.
Reported-by: Brian Carpenter
Fixes#4807Closes#4810
- When creating a directory hierarchy do not error when mkdir fails due
to error EACCESS (13) "access denied".
Some file systems allow for directory traversal; in this case that it
should be possible to create child directories when permission to the
parent directory is restricted.
This is a regression caused by me in f16bed0 (precedes curl-7_61_1).
Basically I had assumed that if a directory already existed it would
fail only with error EEXIST, and not error EACCES. The latter may
happen if the directory exists but has certain restricted permissions.
Reported-by: mbeifuss@users.noreply.github.com
Fixes https://github.com/curl/curl/issues/4796
Closes https://github.com/curl/curl/pull/4797
Previously it would end up with an uninitialized memory buffer that
would lead to a crash or junk getting output.
Added test 1271 to verify.
Reported-by: Brian Carpenter
Closes#4786
Change series of error outputs to use errorf().
Only errors that are due to mistakes in command line option usage should
use helpf(), other types of errors in the tool should rather use
errorf().
Closes#4691
Add support for CURLSSLOPT_NO_PARTIALCHAIN in CURLOPT_PROXY_SSL_OPTIONS
and OS400 package spec.
Also I added the option to the NameValue list in the tool even though it
isn't exposed as a command-line option (...yet?). (NameValue stringizes
the option name for the curl cmd -> libcurl source generator)
Follow-up to 564d88a which added CURLSSLOPT_NO_PARTIALCHAIN.
Ref: https://github.com/curl/curl/pull/4655
- Disable warning C4127 "conditional expression is constant" globally
in curl_setup.h for when building with Microsoft's compiler.
This mainly affects building with the Visual Studio project files found
in the projects dir.
Prior to this change the cmake and winbuild build systems already
disabled 4127 globally for when building with Microsoft's compiler.
Also, 4127 was already disabled for all build systems in the limited
circumstance of the WHILE_FALSE macro which disabled the warning
specifically for while(0). This commit removes the WHILE_FALSE macro and
all other cruft in favor of disabling globally in curl_setup.
Background:
We have various macros that cause 0 or 1 to be evaluated, which would
cause warning C4127 in Visual Studio. For example this causes it:
#define Curl_resolver_asynch() 1
Full behavior is not clearly defined and inconsistent across versions.
However it is documented that since VS 2015 Update 3 Microsoft has
addressed this somewhat but not entirely, not warning on while(true) for
example.
Prior to this change some C4127 warnings occurred when I built with
Visual Studio using the generated projects in the projects dir.
Closes https://github.com/curl/curl/pull/4658
Attempt to unpause a busy read in the CURLOPT_XFERINFOFUNCTION.
When uploading from stdin in non-blocking mode, a delay in reading
the stream (EAGAIN) causes curl to pause sending data
(CURL_READFUNC_PAUSE). Prior to this change, a busy read was
detected and unpaused only in the CURLOPT_WRITEFUNCTION handler.
This change performs the same busy read handling in a
CURLOPT_XFERINFOFUNCTION handler.
Fixes#2051Closes#4599
Reported-by: bdry on github
Starting with this change when doing parallel transfers, without this
option set, curl will prefer to create new transfers multiplexed on an
existing connection rather than creating a brand new one.
--parallel-immediate can be set to tell curl to prefer to use new
connections rather than to wait and try to multiplex.
libcurl-wise, this means that curl will set CURLOPT_PIPEWAIT by default
on parallel transfers.
Suggested-by: Tom van der Woerdt
Closes#4500
Regression from e59371a493 (7.67.0)
Added test 490, 491 and 492 to verify the functionality.
Reported-by: Kamil Dudka
Reported-by: Anderson Sasaki
Fixes#4588Closes#4591
- If server header Retry-After is being used for retry sleep time then
show that value to the user instead of the normal retry sleep time.
This is a follow-up to 640b973 (7.66.0) which changed curl tool so that
the value from Retry-After header overrides other retry timing options.
Closes https://github.com/curl/curl/pull/4498
New option that allows a user to ONLY switch off curl's progress meter
and leave everything else in "talkative" mode.
Reported-by: Piotr Komborski
Fixes#4422Closes#4470
This should again enable crazy-large download ranges of the style
[1-10000000] that otherwise easily ran out of memory starting in 7.66.0
when this new handle allocating scheme was introduced.
Reported-by: Peter Sumatra
Fixes#4393Closes#4438
When looping around the ranges and given URLs to create transfers, all
errors should exit the loop and return. Previously it would keep
looping.
Reported-by: SumatraPeter on github
Bug: #4393Closes#4396
This commit fixes a regression introduced by curl-7_65_3-5-gb88940850.
Detected by tests 2005, 2008, 2009, 2010, 2011, and 2012 with valgrind
and libmetalink enabled.
Closes#4326
Even though it cannot fall-back to a lower HTTP version automatically. The
safer way to upgrade remains via CURLOPT_ALTSVC.
CURLOPT_H3 no longer has any bits that do anything and might be removed
before we remove the experimental label.
Updated the curl tool accordingly to use "--http3".
Closes#4197
Repeatedly we see problems where using curl_multi_wait() is difficult or
just awkward because if it has no file descriptor to wait for
internally, it returns immediately and leaves it to the caller to wait
for a small amount of time in order to avoid occasional busy-looping.
This is often missed or misunderstood, leading to underperforming
applications.
This change introduces curl_multi_poll() as a replacement drop-in
function that accepts the exact same set of arguments. This function
works identically to curl_multi_wait() - EXCEPT - for the case when
there's nothing to wait for internally, as then this function will by
itself wait for a "suitable" short time before it returns. This
effectiely avoids all risks of busy-looping and should also make it less
likely that apps "over-wait".
This also changes the curl tool to use this funtion internally when
doing parallel transfers and changes curl_easy_perform() to use it
internally.
Closes#4163
As the plan has been laid out in DEPRECATED. Update docs accordingly and
verify in test 1174. Now requires the option to be set to allow HTTP/0.9
responses.
Closes#4191
... to avoid integer overflows later when multiplying with 1000 to
convert seconds to milliseconds.
Added test 1269 to verify.
Reported-by: Jason Lee
Closes#4166
USe configure --with-ngtcp2 or --with-quiche
Using either option will enable a HTTP3 build.
Co-authored-by: Alessandro Ghedini <alessandro@ghedini.me>
Closes#3500
This is done by making sure each individual transfer is first added to a
linked list as then they can be performed serially, or at will, in
parallel.
Closes#3804
... as larger values would rather indicate something silly (and could
potentially cause buffer problems).
Reported-by: pendrek at hackerone
Closes#4114
Daniel Stenberg