Commit Graph

682 Commits

Author SHA1 Message Date
Daniel Stenberg 265b14d6b3
metalink: remove
Warning: this will make existing curl command lines that use metalink to
stop working.

Reasons for removal:

1. We've found several security problems and issues involving the
   metalink support in curl. The issues are not detailed here. When
   working on those, it become apparent to the team that several of the
   problems are due to the system design, metalink library API and what
   the metalink RFC says. They are very hard to fix on the curl side
   only.

2. The metalink usage with curl was only very briefly documented and was
   not following the "normal" curl usage pattern in several ways, making
   it surprising and non-intuitive which could lead to further security
   issues.

3. The metalink library was last updated 6 years ago and wasn't so
   active the years before that either. An unmaintained library means
   there's a security problem waiting to happen. This is probably reason
   enough.

4. Metalink requires an XML parsing library, which is complex code (even
   the smaller alternatives) and to this day often gets security
   updates.

5. Metalink is not a widely used curl feature. In the 2020 curl user
   survey, only 1.4% of the responders said that they'd are using it. In
   2021 that number was 1.2%. Searching the web also show very few
   traces of it being used, even with other tools.

6. The torrent format and associated technology clearly won for
   downloading large files from multiple sources in parallel.

Cloes #7176
2021-06-07 08:14:25 +02:00
Daniel Stenberg b193fd3bbd
runtests: enable 'hyper mode' only for HTTP tests
The 'hyper mode' makes line-ending checks work in the test suite for
when hyper is used. Now it also requires that HTTP or HTTPS are
mentioned as keywords to be enabled so that it doesn't wrongly adjusts
tests for other protocols.

This makes test 271 (TFTP) work again in hyper enabled builds.

Closes #7185
2021-06-03 23:08:47 +02:00
Daniel Stenberg 3c0f4622cd
runtests: parse data/Makefile.inc instead of using make
The warning about missing entries in that file then doesn't require that
the Makefile has been regenerated which was confusing.

The scan for the test num is a little more error prone than before
(since now it doesn't actually verify that it is legitimate Makefile
syntax), but I think it is good enough.

Closes #7177
2021-06-02 23:21:18 +02:00
Daniel Stenberg 6fc805d0c1
test972: verify the json output with jsonlint
Make sure one of the azure jobs has jsonlint installed so that the test
runs there.

Ref: #6905
2021-04-17 11:03:40 +02:00
Marc Hoersken 311c31ec8e
runtests.pl: kill processes locking test log files
Introduce a new runtests.pl command option: -rm

For now only required and implemented for Windows.
Ignore stunnel logs due to long running processes.

Requires Sysinternals handle[64].exe to be on PATH.

Reviewed-by: Jay Satiro

Ref: #6058
Closes #6179
2021-03-01 20:19:01 +01:00
Viktor Szakats bd6b54ba1f
tests: disable .curlrc in more environments
by also setting CURL_HOME and XDG_CONFIG_HOME envvars to the local
directory.

Reviewed-by: Daniel Stenberg
Fixes #6595
Closes #6596
2021-02-11 22:06:02 +00:00
Stephan Szabo 1269c80af1 file: Support unicode urls on windows
Closes https://github.com/curl/curl/pull/6501
2021-02-09 02:45:51 -05:00
Fabian Keil 203dc00718
runtests.pl: add %TESTNUMBER variable to make copying tests more convenient 2021-02-09 08:16:40 +01:00
Fabian Keil 8ad31a8c6e
runtests.pl: add an -o option to change internal variables
runtests.pl has lots of internal variables one might want to
change in certain situations, but adding a dedicated option
for every single one of them isn't practical.

Usage:
./runtests.pl -o TESTDIR=$privoxy_curl_test_dir -o HOSTIP=10.0.0.1 ...
2021-02-09 00:28:29 +01:00
Fabian Keil e6d55f647d
runtests.pl: cleanups
- show the summarized test result in the last line of the report
- do not use $_ after mapping it to a named variable
  Doing that makes the code harder to follow.
- log the restraints sorted by the number of their occurrences
- fix language when logging restraints that only occured once
- let runhttpserver() use $TESTDIR instead of $srcdir
  ... so it works if a non-default $TESTDIR is being used.
2021-02-09 00:28:29 +01:00
Fabian Keil 3f0bef2b53
runtests.pl: add an -E option to specify an exclude file
It can contain additional restraints for test numbers,
keywords and tools.

The idea is to let third parties like the Privoxy project
distribute an exclude file with their tarballs that specifies
which curl tests are not expected to work when using Privoxy
as a proxy, without having to fork the whole curl test suite.

The syntax could be changed to be extendable and maybe
more closely reflect the "curl test" syntax. Currently
it's a bunch of lines like these:

test:$TESTNUMBER:Reason why this test with number $TESTNUMBER should be skipped
keyword:$KEYWORD:Reason why tests whose keywords contain the $KEYWORD should be skipped
tool:$TOOL:Reason why tests with tools that contain $TOOL should be skipped

To specify multiple $TESTNUMBERs, $KEYWORDs and $TOOLs
on a single line, split them with commas.
2021-02-09 00:28:28 +01:00
Fabian Keil b47ee58fb7
runtests.pl: add -L parameter to require additional perl libraries
This is useful to change the behaviour of the script without
having to modify the file itself, for example to use a custom
compareparts() function that ignores header differences that
are expected to occur when an external proxy is being used.

Such differences are proxy-specific and thus the modifications
should be maintained together with the proxy.
2021-02-09 00:28:28 +01:00
Fabian Keil 4421359e77
runtests.pl: add a -P option to specify an external proxy
... that should be used when executing the tests.

The assumption is that the proxy is an HTTP proxy.

This option should be used together with -L to provide
a customized compareparts() version that knows which
proxy-specific header differences should be ignored.

This option doesn't work for all test types yet.
2021-02-09 00:28:28 +01:00
Daniel Stenberg f451206266
runtests: preprocess DISABLED to allow conditionals
... with this function provided, we can disable tests for specific
environments and setups directly within this file.

Closes #6477
2021-01-19 10:04:41 +01:00
Daniel Stenberg bbfad7e8a1
runtests: turn preprocessing into a separate function
... and remove all other variable substitutions as they're now done once
and for all in the preprocessor.
2021-01-19 10:04:41 +01:00
Fabian Keil 4f61fd87b2
misc: fix typos
Bug: https://curl.se/mail/lib-2021-01/0063.html
Closes #6434
2021-01-11 15:12:25 +01:00
Daniel Stenberg 9d7281ff17
global_init: debug builds allocates a byte in init
... to make build tools/valgrind warn if no curl_global_cleanup is
called.

This is conditionally only done for debug builds with the env variable
CURL_GLOBAL_INIT set.

Closes #6410
2021-01-06 15:13:45 +01:00
Emil Engler 725ec470e2
language: s/behaviour/behavior/g
We currently use both spellings the british "behaviour" and the american
"behavior". However "behavior" is more used in the project so I think
it's worth dropping the british name.

Closes #6395
2021-01-02 23:35:59 +01:00
Daniel Stenberg 48b85c46f1
runtests: add support for gophers, gopher over TLS 2020-12-15 12:58:19 +01:00
Daniel Stenberg 34bb59a868
runtests: add 'wakeup' as a feature 2020-12-11 17:42:17 +01:00
Daniel Stenberg 3e92799171
runtests.pl: fix "uninitialized value" warning
follow-up to e12825c642
2020-12-11 13:53:23 +01:00
Daniel Stenberg e12825c642
runtests: add support for %if [feature] conditions
... to make tests run differently or expect different results depending
on what features that are present or not in curl.

Bonus: initial minor 'Hyper' awareness but nothing is using that yet

Closes #6304
2020-12-11 13:16:02 +01:00
Daniel Stenberg 082422b189 runtests: make 'c-ares' a "feature" to depend on
... also added to the docs.
2020-11-26 17:24:24 +01:00
Daniel Stenberg 4d2f800677
curl.se: new home
Closes #6172
2020-11-04 23:59:47 +01:00
Daniel Stenberg 7385610d0c
hsts: add support for Strict-Transport-Security
- enable in the build (configure)
- header parsing
- host name lookup
- unit tests for the above
- CI build
- CURL_VERSION_HSTS bit
- curl_version_info support
- curl -V output
- curl-config --features
- CURLOPT_HSTS_CTRL
- man page for CURLOPT_HSTS_CTRL
- curl --hsts (sets CURLOPT_HSTS_CTRL and works with --libcurl)
- man page for --hsts
- save cache to disk
- load cache from disk
- CURLOPT_HSTS
- man page for CURLOPT_HSTS
- added docs/HSTS.md
- fixed --version docs
- adjusted curl_easy_duphandle

Closes #5896
2020-11-03 16:08:42 +01:00
Daniel Stenberg 61630a155e
runtests: show keywords when no tests ran
To help out future debugging, runtests now outputs the list of keywords
when it fails because no tests ran.

Ref: #6120
Closes #6126
2020-10-26 23:05:19 +01:00
Daniel Stenberg 4282d5f9d0
runtests: revert the mistaken edit of $CURL
Regression from c4693adc62
2020-10-19 12:01:00 +02:00
Daniel Stenberg 88bf689f94
runtests: return error if no tests ran
... and make TESTFAIL stand out a little better by adding newlines
before and after.

Reported-by: Marc Hörsken
Issue: #6052
Closes #6053
2020-10-15 15:50:11 +02:00
Sergei Nikulov d8fffd718b
CI/tests: fix invocation of tests for CMake builds
Update appveyor.yml to set env variable TFLAGS and run tests
Remove curly braces due to CMake error (${TFLAGS} -> $TFLAGS)
Move testdeps build to build step (per review comments)

Reviewed-by: Marc Hörsken

Closes #6066
Fixes #6052
2020-10-14 07:03:55 +02:00
Marc Hoersken 0a99281c21
CI/tests: use verification curl for test reporting APIs
Avoid using our own, potentially installed, curl for
the test reporting APIs in case it is broken.

Reviewed-by: Daniel Stenberg

Preparation for #6049
Closes #6063
2020-10-13 06:57:02 +02:00
Marc Hoersken 1101fbbf49
runtests.pl: use $LIBDIR variable instead of hardcoded path
Reviewed-by: Daniel Stenberg
Closes #6051
2020-10-08 21:36:43 +02:00
Daniel Stenberg b8e4d1cb0a
runtests: add %repeat[]% for test files
... and use this new keywords in all the test files larger than 50K to reduce
their sizes and make them a lot easier to read and understand.

Closes #6040
2020-10-04 17:16:53 +02:00
Daniel Stenberg e6b21d422e
runtests: provide curl's version string as %VERSION for tests
... so that we can check HTTP requests for User-Agent: curl/%VERSION

Update 600+ test cases accordingly.

Closes #6037
2020-10-02 22:54:23 +02:00
Daniel Stenberg 2dd53e82f1
runtests: allow generating a binary sequence from hex 2020-10-02 09:35:38 +02:00
Daniel Stenberg c4693adc62
imap: make imap_send use dynbuf for the send buffer management
Reuses the buffer and thereby reduces number of mallocs over a transfer.

Closes #6010
2020-09-25 08:35:01 +02:00
Daniel Stenberg b83947c8df
runtests: allow creating files without newlines
Closes #5946
2020-09-14 08:32:45 +02:00
Daniel Stenberg 5620d2cc78
curl: add --output-dir
Works with --create-dirs and with -J

Add test 3008, 3009, 3011, 3012 and 3013 to verify.

Closes #5637
2020-08-24 22:41:37 +02:00
Daniel Stenberg d71ac6711a
runtests: make cleardir() erase dot files too
Because test cases might use dot files.

Closes #5838
2020-08-23 17:45:41 +02:00
Daniel Stenberg 327bd5d026
runtests: avoid 'fail to start' repeated messages in attempt loops
Closes #5834
2020-08-21 17:16:05 +02:00
Daniel Stenberg 15f76a50ee
runtests: clear pid variables when failing to start a server
... as otherwise the parent doesn't detect the failure and believe it
actually worked to start.

Reported-by: Christian Weisgerber
Bug: https://curl.haxx.se/mail/lib-2020-08/0018.html
Closes #5834
2020-08-21 17:15:59 +02:00
Daniel Stenberg 2429f45a97
TLS naming: fix more Winssl and Darwinssl leftovers
The CMake option is now called CMAKE_USE_SCHANNEL

The winbuild flag is USE_SCHANNEL

The CI jobs and build scripts only use the new names and the new name
options

Tests now require 'Schannel' (when necessary)

Closes #5795
2020-08-08 00:19:21 +02:00
Daniel Stenberg 82ed83ae30
runtests: run the DICT server on a random port number
Removed support for -b (base port number)

Closes #5783
2020-08-07 10:23:15 +02:00
Daniel Stenberg 337a28d810
runtests: move the TELNET server to a dynamic port
Rename the port variable to TELNETPORT to better match the existing
pattern.

Closes #5785
2020-08-06 23:47:08 +02:00
Daniel Stenberg 70999e6034
runtests: move the smbserver to use a dynamic port number
Closes #5782
2020-08-05 22:50:03 +02:00
Daniel Stenberg c8ec04065a
runtests: run the http2 tests on a random port number
Closes #5779
2020-08-05 18:14:25 +02:00
Daniel Stenberg 06aa77ae33
runtests: move the gnutls-serv tests to a dynamic port
Affects test 320, 321, 322 and 324.

Closes #5778
2020-08-05 14:44:38 +02:00
Daniel Stenberg 424092e70f
runtests: support dynamicly base64 encoded sections in tests
This allows us to make test cases to use base64 at run-time and still
use and verify information determined at run-time, such as the IMAP test
server's port number in test 842.

This change makes 12 tests run again that basically never ran since we
moved to dynamic port numbers.

ftpserver.pl is adjusted to load test instructions and test number from
the preprocessed test file.

FILEFORMAT.md now documents the new base64 encoding syntax.

Reported-by: Marcel Raad
Fixes #5761
Closes #5775
2020-08-04 18:12:42 +02:00
Marcel Raad 5c2728eb42
runtests.pl: treat LibreSSL and BoringSSL as OpenSSL
This makes the tests that require the OpenSSL feature also run for
those two compatible libraries.

Closes https://github.com/curl/curl/pull/5762
2020-08-02 10:32:00 +02:00
Gilles Vollant e13357b14b
content_encoding: add zstd decoding support
include zstd curl patch for Makefile.m32 from vszakats
and include Add CMake support for zstd from Peter Wu

Helped-by: Viktor Szakats
Helped-by: Peter Wu
Closes #5453
2020-07-12 18:11:37 +02:00
Daniel Stenberg e2a7a6bb9e
tests: add https-proxy support to the test suite
Initial test 1630 added with basic HTTPS-proxy use. HTTPS-proxy is like
HTTP proxy but with a full TLS connection to the proxy.

Closes #5399
2020-05-14 16:21:31 +02:00