If libcurl is built with Unicode support for Windows then it is assumed
the filename string is Unicode in UTF-8 encoding and it is converted to
UTF-16 to be passed to the wide character version of the respective
function (eg wstat). However the filename string may actually be in the
local encoding so, even if it successfully converted to UTF-16, if it
could not be stat/accessed then try again using the local code page
version of the function (eg wstat fails try stat).
We already do this with fopen (ie wfopen fails try fopen), so I think it
makes sense to extend it to stat and access functions.
Closes https://github.com/curl/curl/pull/6514
- Use _imp.lib suffix only for Microsoft's compiler (MSVC).
Prior to this change library suffix _imp.lib was used for the import
library on Windows regardless of compiler.
With this change the other compilers should now use their default
suffix which should be .dll.a.
This change is motivated by the usage of pkg-config on MSYS2.
Indeed, when 'pkg-config --libs libcurl' is used, -lcurl is
passed to ld. The documentation of ld on Windows :
https://sourceware.org/binutils/docs/ld/WIN32.html
lists, in the 'direct linking to a dll' section, the pattern
of the searched import library, and libcurl_imp.lib is not there.
Closes https://github.com/curl/curl/pull/6225
Since the set value then risks getting used like that when the easy
handle is reused by the application.
Also: renamed the struct field from 'ftp_append' to 'remote_append'
since it is also used for SSH protocols.
Closes#6579
... as we ignore it anyway because servers don't report the correct size
and proftpd even blatantly returns a 550.
Updates a set of tests accordingly.
Reported-by: awesomenode on github
Fixes#6564Closes#6565
- Separate ngtcp2_transport_params.
ngtcp2/ngtcp2@05d7adc made ngtcp2_transport_params separate from
ngtcp2_settings.
ngtcp2 master is required to build curl with http3 support.
Closes#6554
- Add support services without region and service prefixes in
the URL endpoint (ex. Min.IO, GCP, Yandex Cloud, Mail.Ru Cloud Solutions, etc)
by providing region and service parameters via aws-sigv4 option.
- Add [:region[:service]] suffix to aws-sigv4 option;
- Fix memory allocation errors.
- Refactor memory management.
- Use Curl_http_method instead() STRING_CUSTOMREQUEST.
- Refactor canonical headers generating.
- Remove repeated sha256_to_hex() usage.
- Add some docs fixes.
- Add some codestyle fixes.
- Add overloaded strndup() for debug - curl_dbg_strndup().
- Update tests.
Closes#6524
... because it turns out several servers out there don't actually behave
correctly otherwise in spite of the fact that the SNI field is
specifically said to be case insensitive in RFC 6066 section 3.
Reported-by: David Earl
Fixes#6540Closes#6543
- Update build instructions in packages/DOS/README
- Extend 'VPATH' with 'vquic' and 'vssh'.
- Allow 'Makefile.dist' to build both 'lib' and 'src'.
- Allow using the Windows hosted djgpp cross compiler to build for MSDOS
under Windows.
- 'USE_SSL' -> 'USE_OPENSSL'
- Added a 'link_EXE' macro. Etc, etc.
- Linking 'curl.exe' needs '$(CURLX_CFILES)' too.
- Do not pick-up '../lib/djgpp/*.o' files. Recompile locally.
- Generate a gzipped 'tool_hugehelp.c' if 'USE_ZLIB=1'.
- Remove 'djgpp-clean'
- Adapt to new C-ares directory structure
- Use conditional variable assignments
Clarify the 'conditional variable assignment' in 'common.dj'.
Closes https://github.com/curl/curl/pull/6382
This is a follow-up to 8315343 which several days ago moved the resolver
pointer into the async struct but did not update the code that uses it
when getaddrinfo is not present.
Closes https://github.com/curl/curl/pull/6536
As the info is already stored in the transfer handle anyway, there's no
need to carry around a duplicate buffer for the life-time of the handle.
Closes#6534
... and use 'int' for ports. We don't use 'unsigned short' since -1 is
still often used internally to signify "unknown value" and 0 - 65535 are
all valid port numbers.
Closes#6534
The old function should not be used anywhere anymore (the only remaining
gskit use has to be fixed to instead use Curl_poll or none at all).
The static function version is now called our_select() and is only built
if necessary.
Closes#6531
Readdir data, filenames and attributes are strictly related to the
transfer and not the connection. This also reduces the total size of the
fixed connectdata struct.
Closes#6519
On Windows an error number may be greater than INT_MAX and negative once
cast to int.
The assertion is checked only in debug builds.
Closes https://github.com/curl/curl/pull/6504
... if Curl_doh() returned a NULL, this function gets called anyway as
in a asynch procedure. Then the doh struct pointer is NULL and signifies
an OOM situation.
Follow-up to 6246a1d8c6
- Reorder some internal struct members so that less padding is used.
This is an attempt at saving a bit of space by packing some structs
(using pahole to find the holes) where it might make sense to do
so without losing readability.
I.e., I tried to avoid separating fields that seem grouped
together (like the cwd... fields in struct ftp_conn for instance).
Also abstained from touching fields behind conditional macros as
that quickly can get complicated.
Closes https://github.com/curl/curl/pull/6483
... instead of having it static within the Curl_easy struct. This takes
away 1176 bytes (18%) from the Curl_easy struct that aren't used very
often and instead makes the code allocate it when needed.
Closes#6492
The SOCKS code now uses the generic download buffer for temporary
storage during the connection procedure, instead of having its own
private 600 byte buffer that adds to the connectdata struct size. This
works fine because this point the buffer is allocated but is not use for
download yet since the connection hasn't completed.
This reduces the connection struct size by 22% on a 64bit arch!
The SOCKS buffer needs to be at least 600 bytes, and the download buffer
is guaranteed to never be smaller than 1000 bytes.
Closes#6491
By making the `magic` identifier the same size and at the same place
within the structs (easy, multi, share), libcurl will be able to more
reliably detect and safely error out if an application passes in the
wrong handle to APIs. Easier to detect and less likely to cause crashes
if done.
Such mixups can't be detected at compile-time due to them being
typedefed void pointers - unless `CURL_STRICTER` is defined.
Closes#6484
Since curl's own memory debugging system redefines free() calls to track
and fiddle with memory, it cannot be used on memory allocated by 3rd
party libraries.
Third party libraries SHOULD NOT require free() to release allocated
resources for this reason - and libs can use separate healp allocators
on some systems (like Windows) so free() doesn't necessarily work
anyway.
Filed as an issue with libssh: https://bugs.libssh.org/T268Closes#6481
... in most cases instead of 'struct connectdata *' but in some cases in
addition to.
- We mostly operate on transfers and not connections.
- We need the transfer handle to log, store data and more. Everything in
libcurl is driven by a transfer (the CURL * in the public API).
- This work clarifies and separates the transfers from the connections
better.
- We should avoid "conn->data". Since individual connections can be used
by many transfers when multiplexing, making sure that conn->data
points to the current and correct transfer at all times is difficult
and has been notoriously error-prone over the years. The goal is to
ultimately remove the conn->data pointer for this reason.
Closes#6425
... so that a function can first use MIMEPOST and then set it to NULL to
reset it back to a blank POST.
Added test 584 to verify the fix.
Reported-by: Christoph M. Becker
Fixes#6455Closes#6456
... instead of at end of the DO state. This makes the timer more
accurate for the protocols that use the DOING state (such as FTP), and
simplifies how the function (now called init_perform) is called.
The timer will then include the entire procedure up to PERFORM -
including all instructions for getting the transfer started.
Closes#6454
- During the end-of-headers response phase do not mark the tunnel
complete unless the response body was completely parsed/ignored.
Prior to this change if the entirety of a CONNECT response with chunked
encoding was not received by the time the final header was parsed then
the connection would be marked done prematurely, before all the chunked
data could be read in and ignored (since this is what we do with any
CONNECT response body) and the connection could not be used.
Bug: https://curl.se/mail/lib-2021-01/0033.html
Reported-by: Fabian Keil
Closes https://github.com/curl/curl/pull/6432
When doing a request with a request body expecting a 401/407 back, that
initial request is sent with a zero content-length. Test 177 and more.
Closes#6424
... so that Retry-After and other meta-content can still be used.
Added 1634 to verify. Adjusted test 194 and 281 since --fail now also
includes the header-terminating CRLF in the output before it exits.
Fixes#6408Closes#6409
... to make build tools/valgrind warn if no curl_global_cleanup is
called.
This is conditionally only done for debug builds with the env variable
CURL_GLOBAL_INIT set.
Closes#6410
... and not in the connection setup, as for multiplexed transfers the
connection setup might be skipped and then the transfer would end up
without the set user-agent!
Reported-by: Flameborn on github
Assisted-by: Andrey Gursky
Assisted-by: Jay Satiro
Assisted-by: Mike Gelfand
Fixes#6312Closes#6417
The wolfSSL TLS library defines NO_OLD_TLS in some of their build
configurations and that causes the library to be built without TLS 1.1.
For example if MD5 is explicitly disabled when building wolfSSL then
that defines NO_OLD_TLS and the library is built without TLS 1.1 [1].
Prior to this change attempting to build curl with a wolfSSL that was
built with NO_OLD_TLS would cause a build link error undefined reference
to wolfTLSv1_client_method.
[1]: https://github.com/wolfSSL/wolfssl/blob/v4.5.0-stable/configure.ac#L2366
Bug: https://curl.se/mail/lib-2020-12/0121.html
Reported-by: Julian Montes
Closes https://github.com/curl/curl/pull/6388
When doing HTTP authentication and a port number set with CURLOPT_PORT,
the code would previously have the URL's port number override as if it
had been a redirect to an absolute URL.
Added test 1568 to verify.
Reported-by: UrsusArctos on github
Fixes#6397Closes#6400
We currently use both spellings the british "behaviour" and the american
"behavior". However "behavior" is more used in the project so I think
it's worth dropping the british name.
Closes#6395
Extend the syntax of CURLOPT_RESOLVE strings: allow using a '+' prefix
(similar to the existing '-' prefix for removing entries) to add
DNS cache entries that will time out just like entries that are added
by libcurl itself.
Append " (non-permanent)" to info log message in case a non-permanent
entry is added.
Adjust relevant comments to reflect the new behavior.
Adjust documentation.
Extend unit1607 to test the new functionality.
Closes#6294
Paused transfers should not be stopped due to slow speed even when
CURLOPT_LOW_SPEED_LIMIT is set. Additionally, the slow speed timer is
now reset when the transfer is unpaused - as otherwise it would easily
just trigger immediately after unpausing.
Reported-by: Harry Sintonen
Fixes#6358Closes#6359
... as the socket might be readable all the time when paused and thus
causing a busy-loop.
Reported-by: Harry Sintonen
Reviewed-by: Jay Satiro
Fixes#6356Closes#6357
It is a security process for HTTP.
It doesn't seems to be standard, but it is used by some cloud providers.
Aws:
https://docs.aws.amazon.com/general/latest/gr/signature-version-4.html
Outscale:
https://wiki.outscale.net/display/EN/Creating+a+Canonical+Request
GCP (I didn't test that this code work with GCP though):
https://cloud.google.com/storage/docs/access-control/signing-urls-manually
most of the code is in lib/http_v4_signature.c
Information require by the algorithm:
- The URL
- Current time
- some prefix that are append to some of the signature parameters.
The data extracted from the URL are: the URI, the region,
the host and the API type
example:
https://api.eu-west-2.outscale.com/api/latest/ReadNets
~~~ ~~~~~~~~ ~~~~~~~~~~~~~~~~~~~
^ ^ ^
/ \ URI
API type region
Small description of the algorithm:
- make canonical header using content type, the host, and the date
- hash the post data
- make canonical_request using custom request, the URI,
the get data, the canonical header, the signed header
and post data hash
- hash canonical_request
- make str_to_sign using one of the prefix pass in parameter,
the date, the credential scope and the canonical_request hash
- compute hmac from date, using secret key as key.
- compute hmac from region, using above hmac as key
- compute hmac from api_type, using above hmac as key
- compute hmac from request_type, using above hmac as key
- compute hmac from str_to_sign using above hmac as key
- create Authorization header using above hmac, prefix pass in parameter,
the date, and above hash
Signed-off-by: Matthias Gatto <matthias.gatto@outscale.com>
Closes#5703
It seems current hmac implementation use md5 for the hash,
V4 signature require sha256, so I've added the needed struct in
this commit.
I've added the functions that do the hmac in v4 signature file
as a static function ,in the next patch of the serie,
because it's used only by this file.
Signed-off-by: Matthias Gatto <matthias.gatto@outscale.com>
The linux kernel does not report all ICMP errors back to userspace due
to historical reasons.
IP*_RECVERR sockopt must be turned on to have the correct behaviour
which is to pass all ICMP errors to userspace.
See https://bugzilla.kernel.org/show_bug.cgi?id=202355Closes#6341
When the initial request isn't possible to send in its entirety, the
remainder of request would be delivered to the debug callback as data
and would wrongly be counted internally as body-bytes sent.
Extended test 1295 to verify.
Closes#6328
When failing in TOOFAST, the multi_done() wasn't called so the same
cleanup and handling wasn't done like when it fails in PERFORM, which in
the case of FTP could mean that the control connection wouldn't be
marked as "dead" for the CURLE_ABORTED_BY_CALLBACK case. Which caused
ftp_disconnect() to use it to send "QUIT", which could end up waiting
for a response a long time before giving up!
Reported-by: Tomas Berger
Fixes#6333Closes#6337
This commit introduces a "gophers" handler inside the gopher protocol if
USE_SSL is defined. This protocol is no different than the usual gopher
prococol, with the added TLS encapsulation upon connecting. The protocol
has been adopted in the gopher community, and many people have enabled
TLS in their gopher daemons like geomyidae(8), and clients, like clic(1)
and hurl(1).
I have not implemented test units for this protocol because my knowledge
of Perl is sub-par. However, for someone more knowledgeable it might be
fairly trivial, because the same test that tests the plain gopher
protocol can be used for "gophers" just by adding a TLS listener.
Signed-off-by: parazyd <parazyd@dyne.org>
Closes#6208
The error is shown with infof rather than failf so that the user will
see the extended error message information only in verbose mode, and
will still see the standard CURLE_AUTH_ERROR message. For example:
---
* schannel: InitializeSecurityContext failed: SEC_E_QOP_NOT_SUPPORTED
(0x8009030A) - The per-message Quality of Protection is not supported by
the security package
* multi_done
* Connection #1 to host 127.0.0.1 left intact
curl: (94) An authentication function returned an error
---
Ref: https://github.com/curl/curl/issues/6302
Closes https://github.com/curl/curl/pull/6315
If supported, defer port selection until connect() time
if --interface is given and source port is 0.
Reproducer:
* start fast webserver on port 80
* starve system of ephemeral ports
$ sysctl net.ipv4.ip_local_port_range="60990 60999"
* start a curl/libcurl "crawler"
$curl --keepalive --parallel --parallel-immediate --head --interface
127.0.0.2 "http://127.0.0.[1-254]/file[001-002].txt"
current result:
(possible some successful data)
curl: (45) bind failed with errno 98: Address already in use
result after patch:
(complete success or few connections failing, higlhy depending on load)
Fail only when all the possible 4-tuple combinations are exhausted,
which is impossible to do when port is selected at bind() time becuse
the kernel does not know if socket will be listen()'ed on or connect'ed
yet.
Closes#6295
Valgrind will complain that ssrem buffer usage if not explicit
initialized, hence initialize it to zero.
This completes the change intially started in commit 2c0d721215 ('ftp:
retry getpeername for FTP with TCP_FASTOPEN') where the ssloc buffer has
a similar memset to zero.
Signed-off-by: Hans-Christian Noren Egtvedt <hegtvedt@cisco.com>
Closes#6289
The command line tool also independently sets --ftp-skip-pasv-ip by
default.
Ten test cases updated to adapt the modified --libcurl output.
Bug: https://curl.se/docs/CVE-2020-8284.html
CVE-2020-8284
Reported-by: Varnavas Papaioannou
... as it makes the URL parser accept "very-long-hostname://" as a valid
host name and we don't want that. The parser now only accepts a blank
(no digits) after the colon if the URL starts with a scheme.
Reported-by: d4d on hackerone
Closes#6283
This reverts commit d2a7d7c185.
This commit also reverts the subsequent follow-ups to that commit, which
were all done within windows #ifdefs that are removed in this
change. Marc helped me verify this.
Fixes#6146Closes#6281
In the case of TFO, the remote host name is not resolved at the
connetion time.
For FTP that has lead to missing hostname for the secondary connection.
Therefore the name resolution is done at the time, when FTP requires it.
Fixes#6252Closes#6265Closes#6282
This field needs to be wide enough to hold sockaddr_in6 when
connecting via IPv6. Otherwise, ngtcp2_conn_read_pkt will drop the
packets because of the address mismatch:
I00000022 [...] con ignore packet from unknown path
We can safely assume that struct sockaddr_storage is available, as it
is used in the public interface of ngtcp2.
Closes#6250
The resolve call is done with the right port number, but the subsequent
check used the wrong one, which then could find a previous resolve which
would return and leave the fresh resolve "incomplete" and leaking
memory.
Fixes#6247Closes#6253
file_disconnect() is identical with file_do() except the function header
but as the arguments are unused anyway so why not just return file_do()
directly!
Reviewed-by: Daniel Stenberg
Closes#6249
The function has been removed from common usage. Also removed comment in
gopher.c that still referenced it.
Reported-by: Rikard Falkeborn
Fixes#6242Closes#6243
To reduce use of types that can't be checked at compile time. Also
removes several typecasts.
... and rename the struct field from 'os_specific' to 'tdata'.
Closes#6239
Reviewed-by: Jay Satiro
EVP_MD_CTX_create will allocate memory for the context and returns
NULL in case the allocation fails. Make sure to catch any allocation
failures and exit early if so.
In passing, also move to EVP_DigestInit rather than EVP_DigestInit_ex
as the latter is intended for ENGINE selection which we don't do.
Closes#6224
Reviewed-by: Daniel Stenberg <daniel@haxx.se>
Reviewed-by: Emil Engler <me@emilengler.com>
curl CMake was setting the the EXE flags for static libraries which made
the /manifest:no flag ended up when linking the static library, which is
not a valid flag for lib.exe or llvm-lib.exe and caused llvm-lib to exit
with an error.
The better way to handle this is to make sure that we pass the correct
linker flags to CMAKE_STATIC_LINKER_FLAGS instead.
Reviewed-by: Jakub Zakrzewski
Closes#6195
... instead of the limiting it to just the max input size. As every
input byte can be expanded to 3 output bytes, this could limit the input
string to 2.66 MB instead of the intended 8 MB.
Reported-by: Marc Schlatter
Closes#6192
By setting the speed limit time stamps unconditionally at transfer
start, we can start off a transfer without speed limits and yet allow
them to get set during transfer and have an effect.
Reported-by: Kael1117 on github
Fixes#6162Closes#6184
The line variable will always be NULL in the error path, so remove
the free call since it's pointless.
Closes#6170
Reviewed-by: Daniel Stenberg <daniel@haxx.se>
- enable in the build (configure)
- header parsing
- host name lookup
- unit tests for the above
- CI build
- CURL_VERSION_HSTS bit
- curl_version_info support
- curl -V output
- curl-config --features
- CURLOPT_HSTS_CTRL
- man page for CURLOPT_HSTS_CTRL
- curl --hsts (sets CURLOPT_HSTS_CTRL and works with --libcurl)
- man page for --hsts
- save cache to disk
- load cache from disk
- CURLOPT_HSTS
- man page for CURLOPT_HSTS
- added docs/HSTS.md
- fixed --version docs
- adjusted curl_easy_duphandle
Closes#5896
... when the chunked framing was added, the size of the "body part" of
the data was calculated wrongly so the debug callback would get told a
header chunk a few bytes too big that would also contain the first few
bytes of the request body.
Reported-by: Dirk Wetter
Ref: #6144Closes#6147
Build breaks because the http_proxy field is missing:
vssh/libssh2.c:3119:10: error: 'struct connectdata' has no member named 'http_proxy'
Regression from #6021, shipped in curl 7.73.0
Closes#6125
* fix two build errors due to mismatch between function
declarations and their definitions
* silence two mismatched signs warnings via casts
Approved-by: Daniel Stenberg
Closes#6093
... and remove the objnames scripts they tested. They're not used for
anything anymore so testing them serves no purpose!
Reported-by: Marc Hörsken
Fixes#6080Closes#6081
- Change get_winapi_error() to return the error string in the local
codepage instead of UTF-8 encoding.
Two weeks ago bed5f84 fixed get_winapi_error() to work on xbox, but it
also changed the error string's encoding from local codepage to UTF-8.
We return the local codepage version of the error string because if it
is output to the user's terminal it will likely be with functions which
expect the local codepage (eg fprintf, failf, infof).
This is essentially a partial revert of bed5f84. The support for xbox
remains but the error string is reverted back to local codepage.
Ref: https://github.com/curl/curl/pull/6005
Reviewed-by: Marcel Raad
Closes#6065
Whitespace is spelled without a space between white and space, so
make sure to consistently spell it that way across the codebase.
Closes#6023
Reviewed-by: Daniel Stenberg <daniel@haxx.se>
Reviewed-by: Emil Engler <me@emilengler.com>
There used to be a way to have memdebug fill allocated memory. 9 years
later this has no value there (valgrind and ASAN etc are way better). If
people need to know about it they can have a look at VCS logs.
Closes#5973
This matches what is returned in other TLS backends in the same
situation.
Reviewed-by: Jay Satiro
Reviewed-by: Emil Engler
Follow-up to 5a3efb1
Reported-by: iammrtau on github
Fixes#6003Closes#6018
Added test 348 to verify. Added a 'STOR' command to the test FTP
server to enable test 348. Documented the command in FILEFORMAT.md
Reported-by: Duncan Wilcox
Fixes#6016Closes#6017
When using HTTPS proxy, SSL is used but not in the view of the FTP
protocol handler itself so separate the connection's use of SSL from the
FTP control connection's sue.
Reported-by: Mingtao Yang
Fixes#5523Closes#6006
... reuses the same dynamic buffer instead of doing repeated malloc/free
cycles.
Test case 100 (FTP dir list PASV) does 7 fewer memory allocation calls
after this change in my test setup (132 => 125), curl 7.72.0 needed 140
calls for this.
Test case 103 makes 9 less allocations now (130). Down from 149 in
7.72.0.
Closes#6004
... by introducing a printf() function that appends directly into a
dynbuf: Curl_dyn_vprintf(). This avoids the mandatory extra malloc so if
the buffer is already big enough it can just printf directly into it.
Since this less-malloc version requires tthe use of a library internal
printf function, we only provide this version when building libcurl and
not for the dynbuf code that is used when building the curl tool.
Closes#5998
... as that counter is subsequently used to detect if nothing was
returned from the peer. This made curl return CURLE_OK when it should
have returned CURLE_GOT_NOTHING.
Fixes#5992
Reported-by: Tom van der Woerdt
Closes#5994
Fixed two return code mixups. CURLE_UNKNOWN_OPTION is saved for when the
option is, yeah, not known. Clarified this in the setopt man page too.
Closes#5993
These two files were always tightly connected and it was hard to
understand what went into which. This also allows us to make the
ftpsend() function static (moved from ftp.c).
Removed security.c
Renamed curl_sec.h to krb5.h
Closes#5987
By avoiding an unnecessary error check and the temp use of the tm
struct, the time2epoch conversion function gets a little bit faster.
When repeating test 517, the updated version is perhaps 1% faster (on
one particular build on one particular architecture).
Closes#5985
