1
0
mirror of https://github.com/moparisthebest/curl synced 2024-12-24 17:18:48 -05:00
Commit Graph

4684 Commits

Author SHA1 Message Date
Jay Satiro
65ca229461 tool_writeout: refactor write-out and write-out json
- Deduplicate the logic used by write-out and write-out json.

Rather than have separate writeLong, writeString, etc, logic for
each of write-out and write-out json instead have respective shared
functions that can output either format and a 'use_json' parameter to
indicate whether it is json that is output.

This will make it easier to maintain. Rather than have to go through
two sets of logic now we only have to go through one.

- Support write-out %{errormsg} and %{exitcode} in json.

- Clarify in the doc that %{exitcode} is the exit code of the transfer.

Prior to this change it just said "The numerical exitcode" which
implies it's the exit code of the tool, and it's not necessarily that.

Closes https://github.com/curl/curl/pull/6544
2021-02-09 02:48:27 -05:00
Stephan Szabo
1269c80af1 file: Support unicode urls on windows
Closes https://github.com/curl/curl/pull/6501
2021-02-09 02:45:51 -05:00
Fabian Keil
461e3ef680
tests/server: Bump MAX_TAG_LEN to 200
This is useful for tests containing HTML inside of <data> sections.
For <img> tags it's not uncommon to be longer than the previous
limit of 79 bytes.

An example of a previously problem-causing tag is:
<img src="http://config.privoxy.org/send-banner?type=auto" border="0" title="Killed-http://www.privoxy.org/images/privoxy.png-by-size" width="88" height="31">
which is needed for a Privoxy test for the banners-by-size filter.

Previously it caused server failures like:
12:29:05.786961 ====> Client connect
12:29:05.787116 accept_connection 3 returned 4
12:29:05.787194 accept_connection 3 returned 0
12:29:05.787285 Read 119 bytes
12:29:05.787345 Process 119 bytes request
12:29:05.787407 Got request: GET /banners-by-size/9 HTTP/1.1
12:29:05.787464 Requested test number 9 part 0
12:29:05.787686 getpart() failed with error: -2
12:29:05.787744 - request found to be complete (9)
12:29:05.787912 getpart() failed with error: -2
12:29:05.788048 Wrote request (119 bytes) input to log/server.input
12:29:05.788157 Send response test9 section <data>
12:29:05.788443 getpart() failed with error: -2
12:29:05.788498 instructed to close connection after server-reply
12:29:05.788550 ====> Client disconnect 0
12:29:05.871448 exit_signal_handler: 15
12:29:05.871714 signalled to die
12:29:05.872040 ========> IPv4 sws (port 21108 pid: 51758) exits with signal (15)
2021-02-09 08:16:40 +01:00
Fabian Keil
e11e5b2d46
tests/badsymbols.pl: when opening '$incdir' fails include it in the error message 2021-02-09 08:16:40 +01:00
Fabian Keil
e6f4ab3477
runtests.1: document -o, -P, -L, and -E 2021-02-09 08:16:40 +01:00
Fabian Keil
203dc00718
runtests.pl: add %TESTNUMBER variable to make copying tests more convenient 2021-02-09 08:16:40 +01:00
Fabian Keil
8ad31a8c6e
runtests.pl: add an -o option to change internal variables
runtests.pl has lots of internal variables one might want to
change in certain situations, but adding a dedicated option
for every single one of them isn't practical.

Usage:
./runtests.pl -o TESTDIR=$privoxy_curl_test_dir -o HOSTIP=10.0.0.1 ...
2021-02-09 00:28:29 +01:00
Fabian Keil
e6d55f647d
runtests.pl: cleanups
- show the summarized test result in the last line of the report
- do not use $_ after mapping it to a named variable
  Doing that makes the code harder to follow.
- log the restraints sorted by the number of their occurrences
- fix language when logging restraints that only occured once
- let runhttpserver() use $TESTDIR instead of $srcdir
  ... so it works if a non-default $TESTDIR is being used.
2021-02-09 00:28:29 +01:00
Fabian Keil
3f0bef2b53
runtests.pl: add an -E option to specify an exclude file
It can contain additional restraints for test numbers,
keywords and tools.

The idea is to let third parties like the Privoxy project
distribute an exclude file with their tarballs that specifies
which curl tests are not expected to work when using Privoxy
as a proxy, without having to fork the whole curl test suite.

The syntax could be changed to be extendable and maybe
more closely reflect the "curl test" syntax. Currently
it's a bunch of lines like these:

test:$TESTNUMBER:Reason why this test with number $TESTNUMBER should be skipped
keyword:$KEYWORD:Reason why tests whose keywords contain the $KEYWORD should be skipped
tool:$TOOL:Reason why tests with tools that contain $TOOL should be skipped

To specify multiple $TESTNUMBERs, $KEYWORDs and $TOOLs
on a single line, split them with commas.
2021-02-09 00:28:28 +01:00
Fabian Keil
b47ee58fb7
runtests.pl: add -L parameter to require additional perl libraries
This is useful to change the behaviour of the script without
having to modify the file itself, for example to use a custom
compareparts() function that ignores header differences that
are expected to occur when an external proxy is being used.

Such differences are proxy-specific and thus the modifications
should be maintained together with the proxy.
2021-02-09 00:28:28 +01:00
Fabian Keil
4421359e77
runtests.pl: add a -P option to specify an external proxy
... that should be used when executing the tests.

The assumption is that the proxy is an HTTP proxy.

This option should be used together with -L to provide
a customized compareparts() version that knows which
proxy-specific header differences should be ignored.

This option doesn't work for all test types yet.
2021-02-09 00:28:28 +01:00
Fabian Keil
124198631b
tests: fixup several tests
missing CRs and modified %hostip

lib556/test556: use a real HTTP version to make test reuse more convenient

make sure the weekday in Date headers matches the date

test61: replace stray "^M" (5e 4d) at the end of a cookie with a '^M' (0d)

Gets the test working with external proxies like Privoxy again.

Closes #6463
2021-02-09 00:28:19 +01:00
Daniel Stenberg
e32601e13a
ftp: avoid SIZE when asking for a TYPE A file
... as we ignore it anyway because servers don't report the correct size
and proftpd even blatantly returns a 550.

Updates a set of tests accordingly.

Reported-by: awesomenode on github
Fixes #6564
Closes #6565
2021-02-04 23:06:03 +01:00
Daniel Stenberg
2605a6fabb
copyright: fix year ranges in need of updates 2021-02-03 08:02:36 +01:00
Dmitry Wagin
796ce293de
http: improve AWS HTTP v4 Signature auth
- Add support services without region and service prefixes in
the URL endpoint (ex. Min.IO, GCP, Yandex Cloud, Mail.Ru Cloud Solutions, etc)
by providing region and service parameters via aws-sigv4 option.
- Add [:region[:service]] suffix to aws-sigv4 option;
- Fix memory allocation errors.
- Refactor memory management.
- Use Curl_http_method instead() STRING_CUSTOMREQUEST.
- Refactor canonical headers generating.
- Remove repeated sha256_to_hex() usage.
- Add some docs fixes.
- Add some codestyle fixes.
- Add overloaded strndup() for debug - curl_dbg_strndup().
- Update tests.

Closes #6524
2021-01-30 23:48:22 +01:00
Daniel Stenberg
f6d13090d4
copyright: fix missing year (range) updates 2021-01-29 14:35:13 +01:00
Daniel Stenberg
553f286a8f
tool_writeout: make %{errormsg} blank for no errors
Closes #6539
2021-01-28 07:55:25 +01:00
Jay Satiro
782143ee81 test410: fix for windows
- Pass the very long request header via file instead of command line.

Prior to this change the 49k very long request header string was passed
via command line and on Windows that is too long so it was truncated and
the test would fail (specifically msys CI).

Closes https://github.com/curl/curl/pull/6516
2021-01-25 23:28:02 -05:00
Daniel Stenberg
8335c6417e
hostip: remove conn->data from resolver functions
This also moves the 'async' struct from the connectdata struct into the
Curl_easy struct, which seems like a better home for it.

Closes #6497
2021-01-22 08:25:09 +01:00
Fabian Keil
0015e06151
http: get CURLOPT_REQUEST_TARGET working with a HTTP proxy
Added test 1613 to verify.

Closes #6490
2021-01-20 15:21:13 +01:00
Fabian Keil
6fcf22654e
Add test1181: Proxy request with --proxy-header "Connection: Keep-Alive" 2021-01-19 11:41:23 +01:00
Fabian Keil
3bccc182a2
Add test1180: Proxy request with -H "Proxy-Connection: Keep-Alive"
At the moment the test fails as curl sends two Proxy-Connection
headers.
2021-01-19 11:41:23 +01:00
Daniel Stenberg
f451206266
runtests: preprocess DISABLED to allow conditionals
... with this function provided, we can disable tests for specific
environments and setups directly within this file.

Closes #6477
2021-01-19 10:04:41 +01:00
Daniel Stenberg
bbfad7e8a1
runtests: turn preprocessing into a separate function
... and remove all other variable substitutions as they're now done once
and for all in the preprocessor.
2021-01-19 10:04:41 +01:00
Daniel Stenberg
11b2dd86d3
test410: verify HTTPS GET with a 49K request header
skip test 410 for mesalink in the CI as it otherwise hangs "forever"
2021-01-18 16:10:34 +01:00
Daniel Stenberg
215db086e0
lib: pass in 'struct Curl_easy *' to most functions
... in most cases instead of 'struct connectdata *' but in some cases in
addition to.

- We mostly operate on transfers and not connections.

- We need the transfer handle to log, store data and more. Everything in
  libcurl is driven by a transfer (the CURL * in the public API).

- This work clarifies and separates the transfers from the connections
  better.

- We should avoid "conn->data". Since individual connections can be used
  by many transfers when multiplexing, making sure that conn->data
  points to the current and correct transfer at all times is difficult
  and has been notoriously error-prone over the years. The goal is to
  ultimately remove the conn->data pointer for this reason.

Closes #6425
2021-01-17 23:56:09 +01:00
Daniel Stenberg
30e7641d7d
test1918: verify curl_easy_option_by_name() and curl_easy_option_by_id()
... and as a practical side-effect, make sure that the
Curl_easyopts_check() function is asserted in debug builds, which we
want to detect mismatches between the options list in easyoptions.c and
the options in curl.h

Found-by: Gisle Vanem
Bug: 08e8455ddd (commitcomment-45991815)

Closes #6461
2021-01-15 17:03:29 +01:00
Daniel Stenberg
1123a0eb24
mime: make sure setting MIMEPOST to NULL resets properly
... so that a function can first use MIMEPOST and then set it to NULL to
reset it back to a blank POST.

Added test 584 to verify the fix.

Reported-by: Christoph M. Becker

Fixes #6455
Closes #6456
2021-01-15 08:33:27 +01:00
Fabian Keil
4f61fd87b2
misc: fix typos
Bug: https://curl.se/mail/lib-2021-01/0063.html
Closes #6434
2021-01-11 15:12:25 +01:00
Daniel Stenberg
060658f634
test179: use consistent header line endings
... to make "Hyper mode" work better.
2021-01-08 10:57:48 +01:00
Daniel Stenberg
ab525c059e
http: have CURLOPT_FAILONERROR fail after all headers
... so that Retry-After and other meta-content can still be used.

Added 1634 to verify. Adjusted test 194 and 281 since --fail now also
includes the header-terminating CRLF in the output before it exits.

Fixes #6408
Closes #6409
2021-01-06 15:14:40 +01:00
Daniel Stenberg
9d7281ff17
global_init: debug builds allocates a byte in init
... to make build tools/valgrind warn if no curl_global_cleanup is
called.

This is conditionally only done for debug builds with the env variable
CURL_GLOBAL_INIT set.

Closes #6410
2021-01-06 15:13:45 +01:00
Daniel Stenberg
75f7ab28d2
lib/unit tests: add missing curl_global_cleanup() calls 2021-01-06 15:13:45 +01:00
Daniel Stenberg
80c5a838ce
test66: disable with Hyper
...as Hyper doesn't support HTTP/0.9
2021-01-05 17:27:05 +01:00
Daniel Stenberg
82e6d34803
test1633: set appropriate name
"--retry with a 429 response and Retry-After:"
2021-01-04 17:58:05 +01:00
Daniel Stenberg
648712eec1
httpauth: make multi-request auth work with custom port
When doing HTTP authentication and a port number set with CURLOPT_PORT,
the code would previously have the URL's port number override as if it
had been a redirect to an absolute URL.

Added test 1568 to verify.

Reported-by: UrsusArctos on github
Fixes #6397
Closes #6400
2021-01-02 23:40:47 +01:00
Emil Engler
725ec470e2
language: s/behaviour/behavior/g
We currently use both spellings the british "behaviour" and the american
"behavior". However "behavior" is more used in the project so I think
it's worth dropping the british name.

Closes #6395
2021-01-02 23:35:59 +01:00
Olaf Hering
f2ed79d8ee
docs/examples: adjust prototypes for CURLOPT_READFUNCTION
The type of the buffer in curl_read_callback is 'char *', not 'void *'.

Signed-off-by: Olaf Hering <olaf@aepfle.de>
Closes #6392
2020-12-30 22:20:24 +01:00
Daniel Stenberg
b9746575a9
tests/unit: fix empty statements with no effect
... by making macros use "do {} while(0)"
2020-12-29 23:02:43 +01:00
Paul Groke
8324dc8b1a
dns: extend CURLOPT_RESOLVE syntax for adding non-permanent entries
Extend the syntax of CURLOPT_RESOLVE strings: allow using a '+' prefix
(similar to the existing '-' prefix for removing entries) to add
DNS cache entries that will time out just like entries that are added
by libcurl itself.

Append " (non-permanent)" to info log message in case a non-permanent
entry is added.

Adjust relevant comments to reflect the new behavior.

Adjust documentation.

Extend unit1607 to test the new functionality.

Closes #6294
2020-12-29 17:49:36 +01:00
XhmikosR
aadc75626b
misc: assorted typo fixes
Closes #6375
2020-12-26 23:54:25 +01:00
Daniel Stenberg
e99e5ab820
tests/badsymbols.pl: ignore stand-alone single hash lines
Bug: https://curl.se/mail/lib-2020-12/0084.html
Reported-by: Dennis Clarke
Assisted-by: Jay Satiro

Closes #6355
2020-12-22 23:12:10 +01:00
Daniel Stenberg
a93c647de8
examples/libtest: add .checksrc to dist
... so that (auto)builds from tarballs also get the correct instructions.

Fixes #6176
Closes #6353
2020-12-21 17:06:06 +01:00
Daniel Stenberg
58c4b8d636
test: verify new --write-out variables
Extended test 1029 and added 1188
2020-12-21 16:38:53 +01:00
Daniel Stenberg
f7a65979f3
test970: adapted to the new internal order of variables 2020-12-21 16:38:52 +01:00
Matthias Gatto
ebdb5f23cc
tests: add very simple AWS HTTP v4 Signature test
Signed-off-by: Matthias Gatto <matthias.gatto@outscale.com>
2020-12-21 16:28:03 +01:00
Daniel Stenberg
a7696c7343
curl: add --create-file-mode [mode]
This option sets the (octal) mode to use for the remote file when one is
created, using the SFTP, SCP or FILE protocols. When not set, the
default is 0644.

Closes #6244
2020-12-21 10:52:41 +01:00
Jay Satiro
7f170a0f5f badsymbols.pl: Add verbose mode -v
Use -v as the first option to enable verbose mode which will show source
input, extracted symbol and line info. For example:

Source: ./../include/curl/typecheck-gcc.h
Symbol: curlcheck_socket_info(info)
Line #423: #define curlcheck_socket_info(info)                     \

Ref: https://curl.se/mail/lib-2020-12/0084.html

Closes https://github.com/curl/curl/pull/6349
2020-12-20 15:59:18 -05:00
Daniel Stenberg
debf23eead
mqtt: handle POST/PUBLISH without a set POSTFIELDSIZE
Detected by OSS-Fuzz
Bug: https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=28735

Added test 1916 and 1917 to verify.

Closes #6338
2020-12-18 12:54:05 +01:00
Daniel Stenberg
c7dd08ce58
tests: updated tests for Hyper 2020-12-18 09:58:04 +01:00