The libssh2 backend has SSH session associated with the connection but
the callback context is the easy handle, so when a connection gets
attached to a transfer, the protocol handler now allows for a custom
function to get used to set things up correctly.
Reported-by: Michael O'Farrell
Fixes#6898Closes#7078
Also added 'CURL_SMALLSENDS' to make Curl_write() send short packets,
which helped verifying this even more.
Add test 363 to verify.
Reported-by: ustcqidi on github
Fixes#6950Closes#7024
Previously this logic would cap the send to CURL_MAX_WRITE_SIZE bytes,
but for the situations where a larger upload buffer has been set, this
function can benefit from sending more bytes. With default size used,
this does the same as before.
Also changed the storage of the size to an 'unsigned int' as it is not
allowed to be set larger than 2M.
Also added cautions to the man pages about changing buffer sizes in
run-time.
Closes#7022
- Disable auto credentials by default. This is a breaking change
for clients that are using it, wittingly or not.
- New libcurl ssl option value CURLSSLOPT_AUTO_CLIENT_CERT tells libcurl
to automatically locate and use a client certificate for
authentication, when requested by the server.
- New curl tool options --ssl-auto-client-cert and
--proxy-ssl-auto-client-cert map to CURLSSLOPT_AUTO_CLIENT_CERT.
This option is only supported for Schannel (the native Windows SSL
library). Prior to this change Schannel would, with no notification to
the client, attempt to locate a client certificate and send it to the
server, when requested by the server. Since the server can request any
certificate that supports client authentication in the OS certificate
store it could be a privacy violation and unexpected.
Fixes https://github.com/curl/curl/issues/2262
Reported-by: Jeroen Ooms
Assisted-by: Wes Hinsley
Assisted-by: Rich FitzJohn
Ref: https://curl.se/mail/lib-2021-02/0066.html
Reported-by: Morten Minde Neergaard
Closes https://github.com/curl/curl/pull/6673
Both were used for the same purposes and there was no logical separation
between them. Combined, this also saves 16 bytes in less holes in my
test build.
Closes#6798
The Curl_easy pointer struct entry in connectdata is now gone. Just
before commit 215db086e0 landed on January 8, 2021 there were 919
references to conn->data.
Closes#6608
- New libcurl options CURLOPT_DOH_SSL_VERIFYHOST,
CURLOPT_DOH_SSL_VERIFYPEER and CURLOPT_DOH_SSL_VERIFYSTATUS do the
same as their respective counterparts.
- New curl tool options --doh-insecure and --doh-cert-status do the same
as their respective counterparts.
Prior to this change DOH SSL certificate verification settings for
verifyhost and verifypeer were supposed to be inherited respectively
from CURLOPT_SSL_VERIFYHOST and CURLOPT_SSL_VERIFYPEER, but due to a bug
were not. As a result DOH verification remained at the default, ie
enabled, and it was not possible to disable. This commit changes
behavior so that the DOH verification settings are independent and not
inherited.
Ref: https://github.com/curl/curl/pull/4579#issuecomment-554723676
Fixes https://github.com/curl/curl/issues/4578
Closes https://github.com/curl/curl/pull/6597
HTTP auth "accidentally" worked before this cleanup since the code would
always overwrite the connection credentials with the credentials from
the most recent transfer and since HTTP auth is typically done first
thing, this has not been an issue. It was still wrong and subject to
possible race conditions or future breakage if the sequence of functions
would change.
The data.set.str[] strings MUST remain unmodified exactly as set by the
user, and the credentials to use internally are instead set/updated in
state.aptr.*
Added test 675 to verify different credentials used in two requests done
over a reused HTTP connection, which previously behaved wrongly.
Fixes#6542Closes#6545
Rename it to 'httpwant' and make a cloned field in the state struct as
well for run-time updates.
Also: refuse non-supported HTTP versions. Verified with test 129.
Closes#6585
and rename it from 'ftp_list_only' since it is also used for SSH and
POP3. The state is updated internally for 'type=D' FTP URLs.
Added test case 1570 to verify.
Closes#6578
... and make sure the code never updates 'set.prefer_ascii' as it breaks
handle reuse which should use the setting as the user specified it.
Added test 1569 to verify: it first makes an FTP transfer with ';type=A'
and then another without type on the same handle and the second should
then use binary. Previously, curl failed this.
Closes#6578
Since the set value then risks getting used like that when the easy
handle is reused by the application.
Also: renamed the struct field from 'ftp_append' to 'remote_append'
since it is also used for SSH protocols.
Closes#6579
- Add support services without region and service prefixes in
the URL endpoint (ex. Min.IO, GCP, Yandex Cloud, Mail.Ru Cloud Solutions, etc)
by providing region and service parameters via aws-sigv4 option.
- Add [:region[:service]] suffix to aws-sigv4 option;
- Fix memory allocation errors.
- Refactor memory management.
- Use Curl_http_method instead() STRING_CUSTOMREQUEST.
- Refactor canonical headers generating.
- Remove repeated sha256_to_hex() usage.
- Add some docs fixes.
- Add some codestyle fixes.
- Add overloaded strndup() for debug - curl_dbg_strndup().
- Update tests.
Closes#6524
As the info is already stored in the transfer handle anyway, there's no
need to carry around a duplicate buffer for the life-time of the handle.
Closes#6534
... and use 'int' for ports. We don't use 'unsigned short' since -1 is
still often used internally to signify "unknown value" and 0 - 65535 are
all valid port numbers.
Closes#6534
- Reorder some internal struct members so that less padding is used.
This is an attempt at saving a bit of space by packing some structs
(using pahole to find the holes) where it might make sense to do
so without losing readability.
I.e., I tried to avoid separating fields that seem grouped
together (like the cwd... fields in struct ftp_conn for instance).
Also abstained from touching fields behind conditional macros as
that quickly can get complicated.
Closes https://github.com/curl/curl/pull/6483
... instead of having it static within the Curl_easy struct. This takes
away 1176 bytes (18%) from the Curl_easy struct that aren't used very
often and instead makes the code allocate it when needed.
Closes#6492
The SOCKS code now uses the generic download buffer for temporary
storage during the connection procedure, instead of having its own
private 600 byte buffer that adds to the connectdata struct size. This
works fine because this point the buffer is allocated but is not use for
download yet since the connection hasn't completed.
This reduces the connection struct size by 22% on a 64bit arch!
The SOCKS buffer needs to be at least 600 bytes, and the download buffer
is guaranteed to never be smaller than 1000 bytes.
Closes#6491
By making the `magic` identifier the same size and at the same place
within the structs (easy, multi, share), libcurl will be able to more
reliably detect and safely error out if an application passes in the
wrong handle to APIs. Easier to detect and less likely to cause crashes
if done.
Such mixups can't be detected at compile-time due to them being
typedefed void pointers - unless `CURL_STRICTER` is defined.
Closes#6484
... in most cases instead of 'struct connectdata *' but in some cases in
addition to.
- We mostly operate on transfers and not connections.
- We need the transfer handle to log, store data and more. Everything in
libcurl is driven by a transfer (the CURL * in the public API).
- This work clarifies and separates the transfers from the connections
better.
- We should avoid "conn->data". Since individual connections can be used
by many transfers when multiplexing, making sure that conn->data
points to the current and correct transfer at all times is difficult
and has been notoriously error-prone over the years. The goal is to
ultimately remove the conn->data pointer for this reason.
Closes#6425
It is a security process for HTTP.
It doesn't seems to be standard, but it is used by some cloud providers.
Aws:
https://docs.aws.amazon.com/general/latest/gr/signature-version-4.html
Outscale:
https://wiki.outscale.net/display/EN/Creating+a+Canonical+Request
GCP (I didn't test that this code work with GCP though):
https://cloud.google.com/storage/docs/access-control/signing-urls-manually
most of the code is in lib/http_v4_signature.c
Information require by the algorithm:
- The URL
- Current time
- some prefix that are append to some of the signature parameters.
The data extracted from the URL are: the URI, the region,
the host and the API type
example:
https://api.eu-west-2.outscale.com/api/latest/ReadNets
~~~ ~~~~~~~~ ~~~~~~~~~~~~~~~~~~~
^ ^ ^
/ \ URI
API type region
Small description of the algorithm:
- make canonical header using content type, the host, and the date
- hash the post data
- make canonical_request using custom request, the URI,
the get data, the canonical header, the signed header
and post data hash
- hash canonical_request
- make str_to_sign using one of the prefix pass in parameter,
the date, the credential scope and the canonical_request hash
- compute hmac from date, using secret key as key.
- compute hmac from region, using above hmac as key
- compute hmac from api_type, using above hmac as key
- compute hmac from request_type, using above hmac as key
- compute hmac from str_to_sign using above hmac as key
- create Authorization header using above hmac, prefix pass in parameter,
the date, and above hash
Signed-off-by: Matthias Gatto <matthias.gatto@outscale.com>
Closes#5703
When the initial request isn't possible to send in its entirety, the
remainder of request would be delivered to the debug callback as data
and would wrongly be counted internally as body-bytes sent.
Extended test 1295 to verify.
Closes#6328
To reduce use of types that can't be checked at compile time. Also
removes several typecasts.
... and rename the struct field from 'os_specific' to 'tdata'.
Closes#6239
Reviewed-by: Jay Satiro
- enable in the build (configure)
- header parsing
- host name lookup
- unit tests for the above
- CI build
- CURL_VERSION_HSTS bit
- curl_version_info support
- curl -V output
- curl-config --features
- CURLOPT_HSTS_CTRL
- man page for CURLOPT_HSTS_CTRL
- curl --hsts (sets CURLOPT_HSTS_CTRL and works with --libcurl)
- man page for --hsts
- save cache to disk
- load cache from disk
- CURLOPT_HSTS
- man page for CURLOPT_HSTS
- added docs/HSTS.md
- fixed --version docs
- adjusted curl_easy_duphandle
Closes#5896
When using HTTPS proxy, SSL is used but not in the view of the FTP
protocol handler itself so separate the connection's use of SSL from the
FTP control connection's sue.
Reported-by: Mingtao Yang
Fixes#5523Closes#6006
