mirror of
https://github.com/moparisthebest/curl
synced 2024-08-13 17:03:50 -04:00
Daniel Stenberg
parent
d0c77730ed
commit
422f610b40
@ -25,7 +25,7 @@ PDFPAGES = testcurl.pdf runtests.pdf
|
||||
MANDISTPAGES = runtests.1.dist testcurl.1.dist
|
||||
|
||||
EXTRA_DIST = ftpserver.pl httpserver.pl secureserver.pl runtests.pl \
|
||||
getpart.pm FILEFORMAT.md README stunnel.pem memanalyze.pl testcurl.pl \
|
||||
getpart.pm FILEFORMAT.md README.md stunnel.pem memanalyze.pl testcurl.pl \
|
||||
valgrind.pm ftp.pm sshserver.pl sshhelp.pm pathhelp.pm testcurl.1 runtests.1 \
|
||||
serverhelp.pm tftpserver.pl rtspserver.pl directories.pm symbol-scan.pl \
|
||||
CMakeLists.txt mem-include-scan.pl valgrind.supp extern-scan.pl \
|
||||
|
||||
@ -1,53 +1,19 @@
|
||||
_ _ ____ _
|
||||
___| | | | _ \| |
|
||||
/ __| | | | |_) | |
|
||||
| (__| |_| | _ <| |___
|
||||
\___|\___/|_| \_\_____|
|
||||
# The curl Test Suite
|
||||
|
||||
The curl Test Suite
|
||||
# Running
|
||||
|
||||
1. Running
|
||||
1.1 Requires to run
|
||||
1.2 Port numbers used by test servers
|
||||
1.3 Test servers
|
||||
1.4 Run
|
||||
1.5 Shell startup scripts
|
||||
1.6 Memory test
|
||||
1.7 Debug
|
||||
1.8 Logs
|
||||
1.9 Test input files
|
||||
1.10 Code coverage
|
||||
1.11 Remote testing
|
||||
## Requires to run
|
||||
|
||||
2. Numbering
|
||||
2.1 Test case numbering
|
||||
- perl (and a unix-style shell)
|
||||
- python (and a unix-style shell, for SMB and TELNET tests)
|
||||
- python-impacket (for SMB tests)
|
||||
- diff (when a test fails, a diff is shown)
|
||||
- stunnel (for HTTPS and FTPS tests)
|
||||
- OpenSSH or SunSSH (for SCP, SFTP and SOCKS4/5 tests)
|
||||
- nghttpx (for HTTP/2 tests)
|
||||
- nroff (for --manual tests)
|
||||
|
||||
3. Write tests
|
||||
3.1 test data
|
||||
3.2 curl tests
|
||||
3.3 libcurl tests
|
||||
3.4 unit tests
|
||||
|
||||
4. TODO
|
||||
4.1 More protocols
|
||||
4.2 SOCKS auth
|
||||
|
||||
==============================================================================
|
||||
|
||||
1. Running
|
||||
|
||||
1.1 Requires to run
|
||||
|
||||
perl (and a unix-style shell)
|
||||
python (and a unix-style shell, for SMB and TELNET tests)
|
||||
python-impacket (for SMB tests)
|
||||
diff (when a test fails, a diff is shown)
|
||||
stunnel (for HTTPS and FTPS tests)
|
||||
OpenSSH or SunSSH (for SCP, SFTP and SOCKS4/5 tests)
|
||||
nghttpx (for HTTP/2 tests)
|
||||
nroff (for --manual tests)
|
||||
|
||||
1.1.1 Installation of python-impacket
|
||||
### Installation of python-impacket
|
||||
|
||||
The Python-based test servers support both recent Python 2 and 3.
|
||||
You can figure out your default Python interpreter with python -V
|
||||
@ -56,33 +22,33 @@ The curl Test Suite
|
||||
You can use pip or your OS' package manager to install 'impacket'.
|
||||
|
||||
On Debian/Ubuntu the package names are:
|
||||
Python 2: 'python-impacket'
|
||||
Python 3: 'python3-impacket'
|
||||
|
||||
- Python 2: 'python-impacket'
|
||||
- Python 3: 'python3-impacket'
|
||||
|
||||
On FreeBSD the package names are:
|
||||
Python 2: 'py27-impacket'
|
||||
Python 3: 'py37-impacket'
|
||||
|
||||
- Python 2: 'py27-impacket'
|
||||
- Python 3: 'py37-impacket'
|
||||
|
||||
On any system where pip is available:
|
||||
Python 2: 'pip2 install impacket'
|
||||
Python 3: 'pip3 install impacket'
|
||||
|
||||
- Python 2: 'pip2 install impacket'
|
||||
- Python 3: 'pip3 install impacket'
|
||||
|
||||
You may also need to manually install the Python package 'six'
|
||||
as that may be a missing requirement for impacket on Python 3.
|
||||
|
||||
1.2 Port numbers used by test servers
|
||||
### Port numbers used by test servers
|
||||
|
||||
Tests are written to use as few fixed fixed port numbers as possible and all
|
||||
tests should be written to use suitable variables instead of port numbers so
|
||||
that test cases continue to work independent on what port numbers the test
|
||||
servers actually use.
|
||||
All test servers run on "random" port numbers. All tests should be written
|
||||
to use suitable variables instead of fixed port numbers so that test cases
|
||||
continue to work independent on what port numbers the test servers actually
|
||||
use.
|
||||
|
||||
The remaining fixed-port test servers that are still used, use the port
|
||||
range 8890 - 8904 by default, but can be moved with runtests' -b option.
|
||||
See [FILEFORMAT](FILEFORMAT.md) for the port number variables.
|
||||
|
||||
See the FILEFORMAT for a listing of existing port number variables.
|
||||
|
||||
1.3 Test servers
|
||||
### Test servers
|
||||
|
||||
The test suite runs simple FTP, POP3, IMAP, SMTP, HTTP and TFTP stand-alone
|
||||
servers on the ports listed above to which it makes requests. For SSL tests,
|
||||
@ -99,27 +65,28 @@ The curl Test Suite
|
||||
The HTTP server supports listening on a Unix domain socket, the default
|
||||
location is 'http.sock'.
|
||||
|
||||
1.4 Run
|
||||
### Run
|
||||
|
||||
'./configure && make && make test'. This builds the test suite support code
|
||||
`./configure && make && make test`. This builds the test suite support code
|
||||
and invokes the 'runtests.pl' perl script to run all the tests. Edit the top
|
||||
variables of that script in case you have some specific needs, or run the
|
||||
script manually (after the support code has been built).
|
||||
|
||||
The script breaks on the first test that doesn't do OK. Use -a to prevent
|
||||
the script from aborting on the first error. Run the script with -v for more
|
||||
verbose output. Use -d to run the test servers with debug output enabled as
|
||||
well. Specifying -k keeps all the log files generated by the test intact.
|
||||
The script breaks on the first test that doesn't do OK. Use `-a` to prevent
|
||||
the script from aborting on the first error. Run the script with `-v` for
|
||||
more verbose output. Use `-d` to run the test servers with debug output
|
||||
enabled as well. Specifying `-k` keeps all the log files generated by the
|
||||
test intact.
|
||||
|
||||
Use -s for shorter output, or pass test numbers to run specific tests only
|
||||
(like "./runtests.pl 3 4" to test 3 and 4 only). It also supports test case
|
||||
ranges with 'to', as in "./runtests 3 to 9" which runs the seven tests from
|
||||
3 to 9. Any test numbers starting with ! are disabled, as are any test
|
||||
numbers found in the files data/DISABLED or data/DISABLED.local (one per
|
||||
Use `-s` for shorter output, or pass test numbers to run specific tests only
|
||||
(like `./runtests.pl 3 4` to test 3 and 4 only). It also supports test case
|
||||
ranges with 'to', as in `./runtests.pl 3 to 9` which runs the seven tests
|
||||
from 3 to 9. Any test numbers starting with ! are disabled, as are any test
|
||||
numbers found in the files `data/DISABLED` or `data/DISABLED.local` (one per
|
||||
line). The latter is meant for local temporary disables and will be ignored
|
||||
by git.
|
||||
|
||||
When -s is not present, each successful test will display on one line the
|
||||
When `-s` is not present, each successful test will display on one line the
|
||||
test number and description and on the next line a set of flags, the test
|
||||
result, current test sequence, total number of tests to be run and an
|
||||
estimated amount of time to complete the test run. The flags consist of
|
||||
@ -134,7 +101,7 @@ The curl Test Suite
|
||||
m memory
|
||||
v valgrind
|
||||
|
||||
1.5 Shell startup scripts
|
||||
### Shell startup scripts
|
||||
|
||||
Tests which use the ssh test server, SCP/SFTP/SOCKS tests, might be badly
|
||||
influenced by the output of system wide or user specific shell startup
|
||||
@ -150,51 +117,51 @@ The curl Test Suite
|
||||
output of a shell startup script. Locate, cleanup or adjust the shell
|
||||
script.
|
||||
|
||||
1.6 Memory test
|
||||
### Memory test
|
||||
|
||||
The test script will check that all allocated memory is freed properly IF
|
||||
curl has been built with the CURLDEBUG define set. The script will
|
||||
curl has been built with the `CURLDEBUG` define set. The script will
|
||||
automatically detect if that is the case, and it will use the
|
||||
'memanalyze.pl' script to analyze the memory debugging output.
|
||||
|
||||
Also, if you run tests on a machine where valgrind is found, the script will
|
||||
use valgrind to run the test with (unless you use -n) to further verify
|
||||
use valgrind to run the test with (unless you use `-n`) to further verify
|
||||
correctness.
|
||||
|
||||
runtests.pl's -t option will enable torture testing mode, which runs each
|
||||
runtests.pl's `-t` option will enable torture testing mode, which runs each
|
||||
test many times and makes each different memory allocation fail on each
|
||||
successive run. This tests the out of memory error handling code to ensure
|
||||
that memory leaks do not occur even in those situations. It can help to
|
||||
compile curl with CPPFLAGS=-DMEMDEBUG_LOG_SYNC when using this option, to
|
||||
compile curl with `CPPFLAGS=-DMEMDEBUG_LOG_SYNC` when using this option, to
|
||||
ensure that the memory log file is properly written even if curl crashes.
|
||||
|
||||
1.7 Debug
|
||||
### Debug
|
||||
|
||||
If a test case fails, you can conveniently get the script to invoke the
|
||||
debugger (gdb) for you with the server running and the exact same command
|
||||
line parameters that failed. Just invoke 'runtests.pl <test number> -g' and
|
||||
line parameters that failed. Just invoke `runtests.pl <test number> -g` and
|
||||
then just type 'run' in the debugger to perform the command through the
|
||||
debugger.
|
||||
|
||||
1.8 Logs
|
||||
### Logs
|
||||
|
||||
All logs are generated in the log/ subdirectory (it is emptied first in the
|
||||
runtests.pl script). Use runtests.pl -k to force it to keep the temporary
|
||||
files after the test run since successful runs will clean it up otherwise.
|
||||
runtests.pl script). They remain in there after a test run.
|
||||
|
||||
1.9 Test input files
|
||||
### Test input files
|
||||
|
||||
All test cases are put in the data/ subdirectory. Each test is stored in the
|
||||
file named according to the test number.
|
||||
All test cases are put in the `data/` subdirectory. Each test is stored in
|
||||
the file named according to the test number.
|
||||
|
||||
See FILEFORMAT for the description of the test case files.
|
||||
See [FILEFORMAT.md](FILEFORMAT.md) for a description of the test case file
|
||||
format.
|
||||
|
||||
1.10 Code coverage
|
||||
### Code coverage
|
||||
|
||||
gcc provides a tool that can determine the code coverage figures for
|
||||
the test suite. To use it, configure curl with
|
||||
CFLAGS='-fprofile-arcs -ftest-coverage -g -O0'. Make sure you run the normal
|
||||
and torture tests to get more full coverage, i.e. do:
|
||||
gcc provides a tool that can determine the code coverage figures for the
|
||||
test suite. To use it, configure curl with `CFLAGS='-fprofile-arcs
|
||||
-ftest-coverage -g -O0`. Make sure you run the normal and torture tests to
|
||||
get more full coverage, i.e. do:
|
||||
|
||||
make test
|
||||
make test-torture
|
||||
@ -207,7 +174,7 @@ The curl Test Suite
|
||||
The text mode tool gcov may also be used, but it doesn't handle object files
|
||||
in more than one directory very well.
|
||||
|
||||
1.11 Remote testing
|
||||
### Remote testing
|
||||
|
||||
The runtests.pl script provides some hooks to allow curl to be tested on a
|
||||
machine where perl can not be run. The test framework in this case runs on
|
||||
@ -215,63 +182,48 @@ The curl Test Suite
|
||||
system using ssh or some other remote execution method. See the comments at
|
||||
the beginning of runtests.pl for details.
|
||||
|
||||
2. Numbering
|
||||
## Test case numbering
|
||||
|
||||
2.1 Test case numbering
|
||||
|
||||
Test cases used to be numbered by category, but the ranges filled
|
||||
Test cases used to be numbered by category ranges, but the ranges filled
|
||||
up. Subsets of tests can now be selected by passing keywords to the
|
||||
runtests.pl script via the make TFLAGS variable.
|
||||
runtests.pl script via the make `TFLAGS` variable.
|
||||
|
||||
New tests should now be added by finding a free number in
|
||||
tests/data/Makefile.inc.
|
||||
New tests are added by finding a free number in `tests/data/Makefile.inc`.
|
||||
|
||||
3. Write tests
|
||||
## Write tests
|
||||
|
||||
Here's a quick description on writing test cases. We basically have three
|
||||
kinds of tests: the ones that test the curl tool, the ones that build small
|
||||
applications and test libcurl directly and the unit tests that test
|
||||
individual (possibly internal) functions.
|
||||
|
||||
3.1 test data
|
||||
### test data
|
||||
|
||||
Each test has a master file that controls all the test data. What to read,
|
||||
what the protocol exchange should look like, what exit code to expect and
|
||||
what command line arguments to use etc.
|
||||
|
||||
These files are tests/data/test[num] where [num] is described in section 2
|
||||
of this document, and the XML-like file format of them is described in the
|
||||
separate tests/FILEFORMAT document.
|
||||
These files are `tests/data/test[num]` where `[num]` is just a unique
|
||||
identifier described above, and the XML-like file format of them is
|
||||
described in the separate [FILEFORMAT.md](FILEFORMAT.md) document.
|
||||
|
||||
3.2 curl tests
|
||||
### curl tests
|
||||
|
||||
A test case that runs the curl tool and verifies that it gets the correct
|
||||
data, it sends the correct data, it uses the correct protocol primitives
|
||||
etc.
|
||||
|
||||
3.3 libcurl tests
|
||||
### libcurl tests
|
||||
|
||||
The libcurl tests are identical to the curl ones, except that they use a
|
||||
specific and dedicated custom-built program to run instead of "curl". This
|
||||
tool is built from source code placed in tests/libtest and if you want to
|
||||
tool is built from source code placed in `tests/libtest` and if you want to
|
||||
make a new libcurl test that is where you add your code.
|
||||
|
||||
3.4 unit tests
|
||||
### unit tests
|
||||
|
||||
Unit tests are tests in the 13xx sequence and they are placed in tests/unit.
|
||||
There's a tests/unit/README describing the specific set of checks and macros
|
||||
that may be used when writing tests that verify behaviors of specific
|
||||
individual functions.
|
||||
Unit tests are placed in `tests/unit`. There's a tests/unit/README
|
||||
describing the specific set of checks and macros that may be used when
|
||||
writing tests that verify behaviors of specific individual functions.
|
||||
|
||||
The unit tests depend on curl being built with debug enabled.
|
||||
|
||||
4. TODO
|
||||
|
||||
4.1 More protocols
|
||||
|
||||
Add tests for TELNET, LDAP, DICT...
|
||||
|
||||
4.2 SOCKS auth
|
||||
|
||||
SOCKS4/5 test deficiencies - no proxy authentication tests as SSH (the
|
||||
test mechanism) doesn't support them
|
||||
Loading…
Reference in New Issue
Block a user