1
0
mirror of https://github.com/moparisthebest/wget synced 2024-07-03 16:38:41 -04:00
Commit Graph

3417 Commits

Author SHA1 Message Date
Hubert Tarasiuk
6064f21c66 Geolocation support for Metalink resources.
* doc/wget.text: Add information about --preferred-location.
* src/init.c: Add --preferred-location option.
* src/main.c (option_data): Handle --preferred-location argument.
(main): Sort resources based on location if requested.
* src/metalink.c (metalink_res_cmp): Compare based on location if
priority and preference are equal.
* src/options.h (options): Add preferred_location option.
2015-07-20 15:31:06 +02:00
Hubert Tarasiuk
97389a7497 Support at most one file signature. Adapt comments to libmetalink 0.13.
* src/metalink.c (retrieve_from_metalink): Add comment about new
libmetalink version. Do not iterate over signatures - support just one.
2015-07-20 15:31:06 +02:00
Hubert Tarasiuk
225a87d4a2 Move some Metalink-related code from http.c to metalink.c.
* src/http.c: Move find_key_value, has_key, find_key_values.
* src/metalink.c: To here.
* src/metalink.h: Make them non-static and add prototypes here.
2015-07-20 15:31:06 +02:00
Hubert Tarasiuk
92a889b278 Unit test for find_key_values.
* src/http.c: Add test_find_key_values.
* src/test.c (main): Run new test.
* src/test.h: Add test_find_key_values.
2015-07-20 15:31:06 +02:00
Hubert Tarasiuk
1113e78534 Unit test for has_key.
* src/http.c: Add test_has_key.
* src/test.c (main): Run new test.
* src/test.h: Add test_has_key.
2015-07-20 15:31:06 +02:00
Hubert Tarasiuk
70cbd59ed6 Unit test for find_key_value.
* src/http.c: Add test_find_key_value.
* src/test.c (main): Run new test.
* src/test.h: Add test_find_key_value.
2015-07-20 15:31:06 +02:00
Hubert Tarasiuk
0e7aff7623 Test case for Metalink over HTTP.
* testenv/Test-metalink-http.py: New test.
* testenv/Makefile.am: Add to test list.
2015-07-20 15:31:06 +02:00
Hubert Tarasiuk
792dd09a87 Support multiple headers with same name in Python test suite.
* testenv/README: Describe how to use repeated header name.
* testenv/server/http/http_server.py (finish_headers): Send all
values from list if the header value is a Python list.
2015-07-20 15:31:06 +02:00
Hubert Tarasiuk
a4f5ced797 Test case for Metalink in XML.
* testenv/Test-metalink-xml.py: New test.
* testenv/Makefile.am: Add file for automake.
2015-07-20 15:31:06 +02:00
Hubert Tarasiuk
05c30c3b1b Start HTTP test only when calling begin().
* testenv/test/http_test.py: Move self.do_test() from __init__ to
begin().
2015-07-20 15:30:39 +02:00
Hubert Tarasiuk
37b58e3976 Metalink support.
* bootstrap.conf: Add crypto/sha256
* configure.ac: Look for libmetalink and GPGME
* doc/wget.texi: Add --input-metalink and --metalink-over-http
options description.
* po/POTFILES.in: Add metalink.c
* src/Makefile.am: Add new translation unit (metalink.c)
* src/http.c (http_stat): Add metalink field.
(free_stat): Free metalink field.
(find_key_value): Find value of given key in header string.
(has_key): Check if token exists in header string.
(find_key_values): Find all key=value pairs in header string.
(metalink_from_http): Obtain Metalink metadata from HTTP response.
(gethttp): Call metalink_from_http if requested.
(http_loop): Request Metalink metadata from HTTP response if should be.
Fall back to regular download if no Metalink metadata found.
* src/init.c: Add --input-metalink and --metalink-over-http options
* src/main.c (option_data): Handle --input-metalink and
--metalink-over-http cmd arguments.
(print_help): Print --input-metalink option description.
(main): Retrieve files from Metalink file
* src/metalink.c (retrieve_from_metalink): Download files described by
metalink.
(metalink_res_cmp): Comparator for resources priority-sorting.
* src/metalink.h: Create header for metalink.c
(RES_TYPE_SUPPORTED): Define supported resources media.
(DEFAULT_PRI): Default mirror priority for Metalink over HTTP.
(VALID_PRI_RANGE): Valid priority range.
* src/options.h (options): Add input_metalink option and metalink_over_http
options.
* src/utils.c (hex_to_string): Convert binary data to ASCII-hex.
* src/utils.h (hex_to_string): Add prototype.
* src/wget.h: Add metalink-related error enums
Add METALINK_METADATA flag for document type.
2015-07-20 15:30:39 +02:00
Romain Bentz
80303366ae Add NULL value check to fix #45289
* src/recur.c (retrieve_tree): Check return value of url_parse()
2015-07-15 18:10:08 +02:00
Tim Rühsen
bd0ffcf8bc Let HTTPS tests XFAIL when no TLS support configured
* configure.ac: Export WITH_SSL for use in Makefile.am
* testenv/Makefile.am: Add HTTPS tests to XFAIL_TESTS when !WITH_SSL

Reported-by: Ander Juaristi <ajuaristi@gmx.es>
2015-07-14 07:54:03 +02:00
Tim Rühsen
25c9b462bf Change function params to const in src/iri.[ch]
* iri.h, iri.c: Added const attribute for params of parse_charsset(),
	check_encoding_name(), idn_encode(), idn_decode(),
	remote_to_utf8(), set_uri_encoding(), set_content_encoding().
2015-07-01 17:15:10 +02:00
Tim Rühsen
77f5a27e65 Work around a libidn <= 1.30 vulnerability
* src/iri.c: Add _utf8_is_valid() to check UTF-8 sequences before
  passing them to idna_to_ascii_8z().
2015-07-01 17:15:05 +02:00
Ángel González
ae58d8a78b Fix wgetrc filename creation for Windows
* init.c/wgetrc_file_name: Remove obsolete code in WINDOWS code path

Reported-by: Gisle Vanem <gvanem@yahoo.no>
2015-06-27 21:32:48 +02:00
Darshit Shah
58702ffd4f Add valgrind suppression files for HTTPS tests
* testenv/test/base_test.py: Use Valgrind SSL suppressions file for
    tests
    * testenv/valgrind-suppression-ssl, tests/valgrind-suppression-ssl:
    Add new suppression files to suppress OpenSSL errors in valgrind
    * tests/test-proxied-https-auth.px: Use the valgrind SSL
    suppressions file for the test
    * tests/test-proxied-https-auth-keepalive.px: Same
2015-06-16 20:31:00 +05:30
Darshit Shah
103f940950 contrib/check-hard: Indentation and spacing cleanup
* contrib/check-hard: Reduce the amount of text output to the
    screen. Also implement some indentation and whitespace cleanups.
2015-06-15 01:35:53 +05:30
Tim Rühsen
5f0818d9f1 Fix usage of CFLAGS in contrib/check-hard
* contrib/check-hard: Set CFLAGS per command line instead of using export.

'make distcheck' changes CFLAGS. So using ./configure -C together with
exported CFLAGS fails. Setting CFLAGS per command line works smoothly.
2015-06-14 20:30:07 +02:00
Tim Rühsen
c6ac51d5bc Move test_* function protoypes from test.c to test.h
* src/test.c: Remove test_* function prototypes, make tests_run static
* src/test.h: Add test_* function protoypes
2015-06-13 22:34:36 +02:00
Giuseppe Scrivano
fd3a3245eb NEWS: cite --if-modified-since 2015-05-23 14:54:17 +02:00
Giuseppe Scrivano
48acb6693d gnulib: update gnulib 2015-05-23 14:51:52 +02:00
Hubert Tarasiuk
885eaaa214 Include --if-modified-since option in user manual.
* doc/wget.texi: Add --if-modified-since section.
2015-05-22 11:08:30 +02:00
Hubert Tarasiuk
8a8d138dcc Support If-Modified-Since header in timestamping mode.
* src/wget.h: Add IF_MODIFIED_SINCE enum for dt. Add TIMECONV_ERR
enum to uerr_t.
* src/http.c (time_to_rfc1123): Convert time_t do http time.
* src/http.c (initialize_request): Include If-Modified-Since header
if appropriate.
* src/http.c (set_file_timestamp): Separate this code from check_file_output.
* src/http.c (check_file_output): Use set_file_timestamp.
* src/http.c (gethttp): Handle properly 304 return code and 200 if server
ignores If-Modified-Since headers.
* src/http.c (http_loop): Load filename to hstat if condget was requested,
use IF_MODIFIED_SINCE if requested and current timestamp can be obtained.
2015-05-22 11:08:30 +02:00
Hubert Tarasiuk
0e8d2d4251 Add --if-modified-since option
* src/init.c: Add to commands array.
* src/main.c: Add to cmdline_option. Add to help message.
* src/options.h: Add to options struct.
2015-05-22 11:08:30 +02:00
Hubert Tarasiuk
be4f91737a Add test for condget requests.
* testenv/Test-condget.py: the test
* testenv/Makefile.am: add to tests list
2015-05-22 11:08:30 +02:00
Hubert Tarasiuk
901bc98edf Support conditional GET in testenv server.
* src/exc/server_error.py: Add exception for GET to HEAD fallback.
* src/server/http/http_server.py: Do not send body if 304 return
code requested for a file.
2015-05-22 11:08:30 +02:00
Hubert Tarasiuk
e397a48f6a Implement timestamp support for local files in testenv
* testenv/README: Change timestamp format definition
* testenv/conf/local_files.py: Set proper timestamps
2015-05-22 11:08:30 +02:00
Pär Karlsson
83537f2415 Fix undeclared loop variable in Perl test suite
Reported-by: Hubert Tarasiuk <hubert.tarasiuk@gmail.com>
2015-05-20 10:04:07 +02:00
Ander Juaristi
8682c2612f Make sure Wget does not unescape reserved chars.
* testenv/Test-reserved-chars.py: New file.

* testenv/Makefile.am: Added new test Test-reserved-chars.py.

When following redirections, Wget should not unescape the reserved
characters that might appear in target URLs.
2015-05-12 21:24:11 +02:00
Ander Juaristi
b0820d553b Fixed incorrect handling of reserved chars.
* src/iri.c (do_conversion): Call url_unescape_except_reserved,
instead of url_unescape.

* src/url.c (url_unescape_1): New static function.
(url_unescape): Calls url_unescape_1 with mask zero. Preserves
same behavior as before. Only code changes.
(url_unescape_except_reserved): New function.

* src/url.h: Added prototype for url_unescape_except_reserved().

When the locale is US-ASCII, URIs that contain special characters
in them are converted to IRIs according to RFC 3987, section 3.2
"Converting URIs to IRIs".
2015-05-12 21:24:06 +02:00
Darshit Shah
b6b1388fb7 Fix documentation for update_speed_ring()
* progress.c (update_speed_ring): The comment for the function
    incorrectly stated that the function uses thirty samples from the
    past instead of twenty.

    Reported-By: Yi Li <lovelylich@gmail.com>
2015-05-07 11:29:07 +05:30
Darshit Shah
9b1dd6dab8 Remove shadowed variable in http.c
* http.c (gethttp): Rename err to conn_err to prevent shadowed
    variable
2015-05-04 21:45:26 +05:30
Steven Schubiger
1cc835dc5b paramcheck: use explicit quoting for here-docs
* util/paramcheck.pl: Adjust here-docs
2015-05-04 10:13:24 +05:30
Tim Ruehsen
6b8dfe1d6e Fix format specifier warning
* src/utils.c (aprintf): Use %d for int argument
2015-05-03 21:18:47 +02:00
Nikolay Merinov
0e6d6ca963 Fix timestamping and continue behaviour with ftp protocol.
* src/ftp.c (ftp_loop_internal): Add option `force_full_retrieve' that force to
retrieve full file.
(ftp_retrieve_list): Pass `true' as `force_full_retrieve' option to
`ftp_loop_internal' if we want to download file with newer timestamp than local
copy.
2015-05-01 00:28:08 +02:00
Rohit Mathulla
3765a1b266 openssl: Read cert from private key file when needed
* src/openssl.c (ssl_init): Assign opt.cert_{file, type}
  from opt.private_key(_type)
2015-04-27 19:52:18 +02:00
Rohit Mathulla
8654f7e2e7 Fix double free bug in SSL code
* src/openssl.c, src/gnutls.c (ssl_init): Copy options using xstrdup
2015-04-27 19:48:51 +02:00
Hubert Tarasiuk
566696cb82 Single exit point and common cleanup code in gethttp
* src/http.c (gethttp): Common cleanup for type, message,
  req, resp, head.  Single exit point.
2015-04-20 10:54:34 +02:00
Tim Rühsen
bebe462a67 Silence warning in perl test suite
* tests/WgetTests.pm: Use string comparisons for $valgrind variable
2015-04-17 22:42:59 +02:00
Tim Rühsen
c579c7bf1e Check memory allocations in WARC code
* src/warc.c: Remove some memory allocations,
              use xmalloc instead of malloc

Reported-by: Bill Parker <wp02855@gmail.com>
2015-04-17 22:42:59 +02:00
Tim Rühsen
4dde3e200f Add more const usage to function params
* warc.c, warc.h: Add const specifier to several function args
2015-04-17 22:42:59 +02:00
Darshit Shah
8e0dd0d870 PEP8'ify the Python Test Suite
* testenv/conf/{__init__,authentication,files_crawled,
      hook_sample,reject_header,server_files}.py: Aesthetic changes to
      meet Python PEP8 guidelines
    * testenv/exc/{server_error,test_failed}.py: Same
    * testenv/misc/{colour_terminal,wget_file}.py: Same
    * testenv/server/http/http_server.py: Same
    * testenv/test/base_test.py: Same
2015-04-14 10:59:17 +05:30
Ander Juaristi
c6af2fddee Improved test suite documentation
* README.checkout: Added reference to Python. New section
  "Testing and development". Updated information about test suites.
2015-04-13 19:36:39 +02:00
Ander Juaristi
e7e3227b34 Added support for GDB in Python tests
* base_test.py (gen_cmd_line): Check GDB_TESTS environment variable.
2015-04-13 19:32:57 +02:00
Tim Ruehsen
5c5d45ae49 Added GDB support to Perl tests
* WgetTests.pm (run): Check GDB_TESTS environment variable.
2015-04-13 19:29:43 +02:00
Hubert Tarasiuk
f4072e5d0b Add Valgrind suppression for libidn.so at idna_to_ascii_4z
* tests/WgetTests.pm (run): Include suppression file when running Valgrind.
* tests/valgrind-suppressions: Add suppression for idn_to_ascii_4z.
* tests/Makefile.am: Add valgrind-suppressions to EXTRA_DIST.
2015-04-11 23:16:32 +02:00
Ángel González
bef5945202 Remove memory leak in idn_encode.
* src/iri.c (idn_encode): Free buffer from remote_to_utf8
when needed; give meaningful names to variables;
remove excessive comment.
2015-04-11 21:55:17 +02:00
Hubert Tarasiuk
ac40b84ee1 Fix error in free_vec.
* src/utils.c (free_vec): Increment pointer instead of its value.

Reported-by: Gisle Vanem <gvanem@yahoo.no>
2015-04-10 18:06:14 +02:00
Ángel González
45463eaad7 Fix const usage in iri.c
* src/iri.c (remote_to_utf8): Do not qualify with const the output pointer.
(do_conversion): Use the provided input parameter as const.
(idn_encode): casts to remote_to_utf8 parameters are no longer needed.
* src/iri.h: Adjusted remote_to_utf8 prototype.
* src/url.c: It is no longer necessary to cast new_url to const char.
2015-04-10 10:23:01 +02:00