Commit Graph

26706 Commits

Author SHA1 Message Date
Daniel Stenberg 7a90ddf88f
curl: add variables to --write-out
In particular, these ones can help a user to create its own error
message when one or transfers fail.

writeout: add 'onerror', 'url', 'urlnum', 'exitcode', 'errormsg'

onerror - lets a user only show the rest on non-zero exit codes

url - the input URL used for this transfer

urlnum - the numerical URL counter (0 indexed) for this transfer

exitcode - the numerical exit code for the transfer

errormsg - obvious

Reported-by: Earnestly on github
Fixes #6199
Closes #6207
2020-12-21 16:38:48 +01:00
Matthias Gatto ebdb5f23cc
tests: add very simple AWS HTTP v4 Signature test
Signed-off-by: Matthias Gatto <matthias.gatto@outscale.com>
2020-12-21 16:28:03 +01:00
Matthias Gatto e2b2afbeea
docs: add AWS HTTP v4 Signature 2020-12-21 16:28:03 +01:00
Matthias Gatto 6b27df8269
tool: add AWS HTTP v4 Signature support
Signed-off-by: Matthias Gatto <matthias.gatto@outscale.com>
2020-12-21 16:28:03 +01:00
Matthias Gatto eb69797766
http: Make the call to v4 signature
This patch allow to call the v4 signature introduce in previous commit

Signed-off-by: Matthias Gatto <matthias.gatto@outscale.com>
2020-12-21 16:28:03 +01:00
Matthias Gatto 08e8455ddd
http: introduce AWS HTTP v4 Signature
It is a security process for HTTP.

It doesn't seems to be standard, but it is used by some cloud providers.

Aws:
https://docs.aws.amazon.com/general/latest/gr/signature-version-4.html
Outscale:
https://wiki.outscale.net/display/EN/Creating+a+Canonical+Request
GCP (I didn't test that this code work with GCP though):
https://cloud.google.com/storage/docs/access-control/signing-urls-manually

most of the code is in lib/http_v4_signature.c

Information require by the algorithm:
- The URL
- Current time
-  some prefix that are append to some of the signature parameters.

The data extracted from the URL are: the URI, the region,
the host and the API type

example:
https://api.eu-west-2.outscale.com/api/latest/ReadNets
        ~~~ ~~~~~~~~               ~~~~~~~~~~~~~~~~~~~
        ^       ^                          ^
       /         \                        URI
   API type     region

Small description of the algorithm:
- make canonical header using content type, the host, and the date
- hash the post data
- make canonical_request using custom request, the URI,
  the get data, the canonical header, the signed header
  and post data hash
- hash canonical_request
- make str_to_sign using one of the prefix pass in parameter,
  the date, the credential scope and the canonical_request hash
- compute hmac from date, using secret key as key.
- compute hmac from region, using above hmac as key
- compute hmac from api_type, using above hmac as key
- compute hmac from request_type, using above hmac as key
- compute hmac from str_to_sign using above hmac as key
- create Authorization header using above hmac, prefix pass in parameter,
  the date, and above hash

Signed-off-by: Matthias Gatto <matthias.gatto@outscale.com>

Closes #5703
2020-12-21 16:27:50 +01:00
Matthias Gatto d52564bacb
http: add hmac support for sha256
It seems current hmac implementation use md5 for the hash,
V4 signature require sha256, so I've added the needed struct in
this commit.

I've added the functions that do the hmac in v4 signature file
as a static function ,in the next patch of the serie,
because it's used only by this file.

Signed-off-by: Matthias Gatto <matthias.gatto@outscale.com>
2020-12-21 15:26:41 +01:00
Cristian Rodríguez d13179db3e
connect: on linux, enable reporting of all ICMP errors on UDP sockets
The linux kernel does not report all ICMP errors back to userspace due
to historical reasons.

IP*_RECVERR sockopt must be turned on to have the correct behaviour
which is to pass all ICMP errors to userspace.

See https://bugzilla.kernel.org/show_bug.cgi?id=202355

Closes #6341
2020-12-21 15:24:08 +01:00
Daniel Stenberg a7696c7343
curl: add --create-file-mode [mode]
This option sets the (octal) mode to use for the remote file when one is
created, using the SFTP, SCP or FILE protocols. When not set, the
default is 0644.

Closes #6244
2020-12-21 10:52:41 +01:00
Daniel Stenberg e1a4647a42
c-hyper: fix compiler warnings
Identified by clang on windows.

Reported-by: Gisle Vanem
Bug: 58974d25d8

Closes #6351
2020-12-20 23:06:46 +01:00
Daniel Stenberg 6e2392f177
KNOWN_BUGS: Remote recursive folder creation with SFTP
Closes #5204
2020-12-20 22:49:59 +01:00
Jay Satiro 7f170a0f5f badsymbols.pl: Add verbose mode -v
Use -v as the first option to enable verbose mode which will show source
input, extracted symbol and line info. For example:

Source: ./../include/curl/typecheck-gcc.h
Symbol: curlcheck_socket_info(info)
Line #423: #define curlcheck_socket_info(info)                     \

Ref: https://curl.se/mail/lib-2020-12/0084.html

Closes https://github.com/curl/curl/pull/6349
2020-12-20 15:59:18 -05:00
Jay Satiro 4cc115a85b KNOWN_BUGS: Secure Transport disabling hostname validation also disables SNI
That behavior is a limitation of Apple's Secure Transport.

Reported-by: Cory Benfield
Reported-by: Ian Spence
Confirmed-by: Nick Zitzmann

Ref: https://github.com/curl/curl/issues/998

Closes https://github.com/curl/curl/issues/6347
Closes https://github.com/curl/curl/pull/6348
2020-12-20 15:58:26 -05:00
Daniel Stenberg ccbdbe13c4
TODO: alt-svc should fallback if alt-svc doesn't work
Closes #4908
2020-12-18 23:39:22 +01:00
Daniel Stenberg 54db6e9902
travis: restrict the openssl3 job to only run https and ftps tests
... as it runs too long otherwise and the other tests are verified in
other builds anyway.

Closes #6345
2020-12-18 23:08:41 +01:00
Daniel Stenberg abb68c192a
build: repair http disabled but mqtt enabled build
... as the mqtt code reuses the "method" originally used for HTTP.

Closes #6344
2020-12-18 16:09:13 +01:00
Jon Wilkes e2075f2337
cookie: avoid the C1001 internal compiler error with MSVC 14
Fixes #6112
Closes #6135
2020-12-18 14:02:27 +01:00
Daniel Stenberg 8fb9f43f81
RELEASE-NOTES: synced 2020-12-18 13:02:40 +01:00
Daniel Stenberg debf23eead
mqtt: handle POST/PUBLISH without a set POSTFIELDSIZE
Detected by OSS-Fuzz
Bug: https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=28735

Added test 1916 and 1917 to verify.

Closes #6338
2020-12-18 12:54:05 +01:00
Daniel Stenberg 92fe66c510
travis: add CI job for Hyper build 2020-12-18 09:58:04 +01:00
Daniel Stenberg c7dd08ce58
tests: updated tests for Hyper 2020-12-18 09:58:04 +01:00
Daniel Stenberg 58974d25d8
lib: introduce c-hyper for using Hyper
... as an alternative HTTP backend within libcurl.
2020-12-18 09:58:04 +01:00
Daniel Stenberg cd7bc174ce
tool_setopt: provide helper output in debug builds
... for when setopt() returns error.
2020-12-18 09:58:04 +01:00
Daniel Stenberg edc89839be
setopt: adjust to Hyper and disabled HTTP builds 2020-12-18 09:58:04 +01:00
Daniel Stenberg 4daba06762
rtsp: disable if Hyper is used 2020-12-18 09:58:04 +01:00
Daniel Stenberg 34021b804d
getinfo: build with disabled HTTP support 2020-12-18 09:58:03 +01:00
Daniel Stenberg 9211cb2034
version: include hyper version 2020-12-18 09:58:03 +01:00
Daniel Stenberg 8a113ba93c
docs: add HYPER.md 2020-12-18 09:58:03 +01:00
Daniel Stenberg c263e583c2
configure: add --with-hyper
As the first (optional) HTTP backend alternative instead of native

Close #6110
2020-12-18 09:57:51 +01:00
Daniel Stenberg ff4d2c2a05
test1522: add debug tracing
I used this to track down some issues and I figured I could just as well
keep this extra logging in here for future needs.

Closes #6331
2020-12-17 16:55:56 +01:00
Daniel Stenberg a5bc272223
http: show the request as headers even when split-sending
When the initial request isn't possible to send in its entirety, the
remainder of request would be delivered to the debug callback as data
and would wrongly be counted internally as body-bytes sent.

Extended test 1295 to verify.

Closes #6328
2020-12-17 14:00:27 +01:00
Daniel Stenberg 3e17c8ab72
multi: when erroring in TOOFAST state, act as for PERFORM
When failing in TOOFAST, the multi_done() wasn't called so the same
cleanup and handling wasn't done like when it fails in PERFORM, which in
the case of FTP could mean that the control connection wouldn't be
marked as "dead" for the CURLE_ABORTED_BY_CALLBACK case. Which caused
ftp_disconnect() to use it to send "QUIT", which could end up waiting
for a response a long time before giving up!

Reported-by: Tomas Berger
Fixes #6333
Closes #6337
2020-12-17 13:57:27 +01:00
Daniel Stenberg ad338b390b
cmake: enable gophers correctly in curl-config
Closes #6336
2020-12-17 13:56:18 +01:00
Daniel Stenberg 3abfadfc19
test1198/9: add two mqtt publish tests without payload lengths
Closes #6335
2020-12-17 13:55:02 +01:00
Daniel Stenberg 30fc1cc7db
tests/mqttd: extract the client id from the correct offset
Closes #6334
2020-12-17 10:15:31 +01:00
Daniel Stenberg f25112074d
TODO: Prevent terminal injection when writing to terminal
Closes #6150
2020-12-16 23:38:20 +01:00
Daniel Stenberg d16fb4d056
Revert "CI/github: work-around for brew breakage on macOS"
This reverts commit 4cbb17a2cb.

... as the work-around now causes failures.

Closes #6332
2020-12-16 14:37:41 +01:00
Daniel Stenberg 1451f4d2e1
examples: remove superfluous asterisk uses
... for function pointers. Breaks in ancient compilers.
2020-12-16 09:24:16 +01:00
Daniel Stenberg 4dc710b198
RELEASE-NOTES: synced 2020-12-16 08:22:28 +01:00
Daniel Stenberg 354bc95d96
test1272: fix line ending
Follow-up to f24784f914
2020-12-15 17:38:42 +01:00
Daniel Stenberg 98b6aa14d9
URL-SYNTAX: add gophers details 2020-12-15 12:58:19 +01:00
Daniel Stenberg f24784f914
test1272: test gophers 2020-12-15 12:58:19 +01:00
Daniel Stenberg 48b85c46f1
runtests: add support for gophers, gopher over TLS 2020-12-15 12:58:19 +01:00
parazyd a1f06f32b8
gopher: Implement secure gopher protocol.
This commit introduces a "gophers" handler inside the gopher protocol if
USE_SSL is defined. This protocol is no different than the usual gopher
prococol, with the added TLS encapsulation upon connecting. The protocol
has been adopted in the gopher community, and many people have enabled
TLS in their gopher daemons like geomyidae(8), and clients, like clic(1)
and hurl(1).

I have not implemented test units for this protocol because my knowledge
of Perl is sub-par. However, for someone more knowledgeable it might be
fairly trivial, because the same test that tests the plain gopher
protocol can be used for "gophers" just by adding a TLS listener.

Signed-off-by: parazyd <parazyd@dyne.org>

Closes #6208
2020-12-15 12:58:12 +01:00
Daniel Stenberg be8c94da78
TODO: Package curl for Windows in a signed installer
Closes #5424
2020-12-15 12:02:14 +01:00
Daniel Stenberg a259eee99f
mqtt: deal with 0 byte reads correctly
OSS-Fuzz found it
Bug: https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=28676

Closes #6327
2020-12-15 11:38:29 +01:00
Daniel Stenberg c4d88f89a9
BUG-BOUNTY: minor language update
... and remove the wording about entries from before 2019 as the "within
12 months" is still there and covers that.

Closes #6318
2020-12-15 08:57:07 +01:00
Daniel Stenberg bc7ecc71c0
tooĺ_writeout: fix the -w time output units
Fix regression from commit fc813f80e1 (#6248) that changed the unit
to microseconds instead of seconds with fractions

Reported-by: 不确定
Fixes #6321
Closes #6322
2020-12-15 08:09:29 +01:00
Daniel Stenberg 796c068895
quiche: remove fprintf() leftover 2020-12-14 11:19:48 +01:00
Jay Satiro 9f85b986a9 KNOWN_BUGS: SHA-256 digest not supported in Windows SSPI builds
Closes https://github.com/curl/curl/issues/6302
2020-12-14 01:08:15 -05:00