1
0
mirror of https://github.com/moparisthebest/wget synced 2024-07-03 16:38:41 -04:00
Commit Graph

2180 Commits

Author SHA1 Message Date
Tim Rühsen
26fadc55c2 Handle TLS rehandshakes in GnuTLS code
* src/gnutls.c: New static function _do_handshake()
* src/gnutls.c (wgnutls_read_timeout): Handle rehandshake
* src/gnutls.c (wgnutls_write): Handle rehandshake
* src/gnutls.c (ssl_connect_wget): Move handshake code into _do_handshake()

Fixes #46061
2015-09-28 16:18:33 +02:00
Darshit Shah
c387db6451 Do not test for impossible qop value
* http.c (digest_authentication_encode): Wget already errors out if
    qop != "auth". Then it makes no sense to test for qop == "auth-int"
    later on. Currently, Wget does not support the "auth-int" qop value
    and till nobidy requests, it may remain so.
2015-09-22 16:36:40 +05:30
Darshit Shah
12dfc03116 Fix #46024. Support RFC 2069 Digest Authentication
* http.c (digest_authentication_encode): Some servers are still
    using the obsolete RFC 2069 Digest Authentication. Allow Digest
    authentication without the qop parameter for this.

    Reported-by: Andreas Longwitz  <longwitz@incore.de>
2015-09-22 15:41:22 +05:30
Darshit Shah
3ea0beec6f Revert "Disable progress bar when wget is backgrounded (trivial patch)"
This reverts commit e624732563.
2015-09-21 19:41:38 +05:30
Ander Juaristi
f8901af4e0 Added support for FTPS
* doc/wget.texi: updated documentation to reflect the new FTPS functionality.
 * src/ftp-basic.c (ftp_greeting): new function to read the server's greeting.
   (ftp_login): greeting code was previously here. Moved to ftp_greeting to
   support FTPS implicit mode.
   (ftp_auth): wrapper around the AUTH TLS command.
   (ftp_ccc): wrapper around the CCC command.
   (ftp_pbsz): wrapper around the PBSZ command.
   (ftp_prot): wraooer around the PROT command.
 * src/ftp.c (get_ftp_greeting): new static function.
   (init_control_ssl_connection): new static function to start SSL/TLS on the
   control channel.
   (getftp): added hooks to support FTPS commands (RFCs 2228 and 4217).
   (ftp_loop_internal): test for new FTPS error codes.
 * src/ftp.h: new enum 'prot_level' with available FTPS protection levels +
   prototypes of previous functions. New flag for enum 'wget_ftp_fstatus' to track
   whether the data channel has some security mechanism enabled or not.
 * src/gnutls.c (struct wgnutls_transport_context): new field 'session_data'.
   (wgnutls_close): free GnuTLS session data before exiting.
   (ssl_connect_wget): save/resume SSL/TLS session.
 * src/http.c (establish_connection): refactor ssl_connect_wget call.
   (metalink_from_http): take into account SCHEME_FTPS as well.
 * src/init.c, src/main.c, src/options.h: new command line/wgetrc options.
   (main): in recursive downloads, check for SCHEME_FTPS as well.
 * src/openssl.c (struct openssl_transport_context): new field 'sess'.
   (ssl_connect_wget): save/resume SSL/TLS session.
 * src/retr.c (retrieve_url): check new scheme SCHEME_FTPS.
 * src/ssl.h (ssl_connect_wget): refactor. New parameter of type 'int *'.
 * src/url.c. src/url.h: new scheme SCHEME_FTPS.
 * src/wget.h: new FTPS error codes.
 * src/metalink.h: support FTPS scheme.
2015-09-14 10:16:44 +02:00
Christian Neukirchen
e624732563 Disable progress bar when wget is backgrounded (trivial patch)
* src/progress.c (create_image): progress only when in foreground

Sometimes I start wget, but the remote site is too slow, so I rather
want to run it in background, however when I simply use job control
for that, wget will keep spewing the progress bar all over my
terminal.  I have found the SIGHUP/SIGUSR1 feature to redirect output
to a log file, but I think the following small patch is even more
useful, since the progress bar will simply resume when wget is
foregrounded again (also, the final message is still printed to the
terminal in any case):
2015-09-10 10:26:29 +02:00
Hubert Tarasiuk
84b9abbf3c Do not free Metalink structure if not initialized
* src/main.c (main): Move metalink_delete to the conditional block.
2015-09-02 09:17:37 +02:00
Ander Juaristi
ab47d9fa3a Extra debug traces for HSTS.
* src/main.c (load_hsts, save_hsts): added DEBUGP() calls to signal
   reads and saves of the HSTS database file.
2015-09-01 13:50:40 +02:00
Darshit Shah
187edb604a Fix coding style violation in last commit
* http.c (test_parse_range_header): Declare loop variable
    explicitly. Not in gnu99 standard.
2015-08-31 21:04:54 +05:30
Darshit Shah
b06fca60ac Add unit test for parse_content_range() method
* http.c (test_parse_range_header): New function to test the
    function for parsing the HTTP/1.1 Content-Range header.
    * test.[ch]: Same
    * http.c (parse_content_range): Fix parsing code. Fail on scenarios
    mentioned in rfc 7233.
2015-08-30 21:34:32 +05:30
Tim Rühsen
c809398e8c Fix null pointer dereference
* src/metalink.c (gpg_skip_verification):
  Check output_stream before fclose
2015-08-30 14:17:47 +02:00
Tim Rühsen
88a1a79bc1 Fix leaks found by Coverity
* src/http.c (parse_strict_transport_security): Free c_max_age
             (open_output_stream): Fix indentation
* src/iri.c (locale_to_utf8): Free new
2015-08-30 14:10:25 +02:00
Tim Rühsen
398699c438 Fix two leaks foudn by Coverity
* src/http.c (gethttp): Do not leak 'message'.
* src/main.c (format_and_print_line): Do not leak 'line_dup'.
2015-08-29 22:35:29 +02:00
Tim Rühsen
d3504b9261 Fix resource leak discovered by Coverity
* src/retr.c (retrieve_url): Don't leak local_file.
2015-08-29 22:15:34 +02:00
Darshit Shah
6b5acff566 Fix memory leaks in unit-test
* hsts.c (get_hsts_store_filename): Free the homedir value
    (close_hsts_test_store): Actually free the store struct too
    (test_hsts_new_entry): Pass store to close_hsts_test_store()
    (test_hsts_url_rewrite_superdomain): Same
    (test_hsts_url_rewrite_congruent): Same
    (test_hsts_read_database): Same and homedir and store filename
    * http.c (test_parse_content_disposition): Free the returned
    filename
    * url.c (test_append_uri_pathel): Free allocated string
2015-08-29 22:52:49 +05:30
Darshit Shah
5c4489db9b Fix mixed-indentation in http.c
* http.c: Fix mix indentation. Visual change only.
2015-08-29 09:45:13 +05:30
Tim Rühsen
7bed9a6f8f Suppress debug output when strings may contain password
* iri.c (do_conversion): Do not print out converted strings if they
  contain an '@'. That could be an URL with embedded password.

Fixes #45825
2015-08-27 09:55:13 +02:00
Ander Juaristi
d080a70a3a Fix resource leak.
* src/http.c (parse_strict_transport_security): Freed memory to avoid resource leak.
   Comply with GNU coding style.
2015-08-26 17:50:26 +05:30
Jookia
030c3379d1 Clarify that links are being converted.
* src/convert.c: Add 'links in' after 'Converted %d' and 'Converting %s'.
2015-08-21 20:58:55 +02:00
Miquel Llobet
e04c5989ff Fixed #44516 -o- not logging to stdout
src/log.c (log_init): check for hypen on filename, set stdout
2015-08-16 00:20:20 +05:30
Daniele Calore
12bae50b28 Fix #40426: Allow -r -O- only if FILE is regular
* main.c: added check of "-r -O FILE" option combination
    allow only if FILE is a regular file (bug #40426)
2015-08-16 00:16:12 +05:30
Darshit Shah
f71887bbe5 Fix var name conflicts with math.h and wingdi.h
* src/recur.c (reject_reason): Rename all enum members to WG_RR_xx.
    * src/recur.c (retrieve_tree, download_child,
    write_reject_log_reason): Same
2015-08-15 15:43:33 +05:30
Tim Rühsen
075d755696 Fix IP address exposure in FTP code
* src/ftp.c (getftp): Do not use PORT when PASV fails.
* tests/FTPServer.px: Add pasv_not_supported server flag.
* tests/Makefile.am: Add Test-ftp-pasv-not-supported.px
* tests/Test-ftp-pasv-not-supported.px: New test

Fix IP address exposure when automatically falling back from
passive mode to active mode (using the PORT command). A behavior that
may be used to expose a client's privacy even when using a proxy.
2015-08-11 17:38:33 +02:00
Tim Rühsen
7578e47d49 Fix C89 compliancy in HSTS test code
* src/hsts.c (test_hsts_new_entry):
  Move variable assignment before code
2015-08-07 14:03:00 +02:00
Tim Rühsen
3a708f7ef8 Fix C89 compliancy in latest code
* src/recur.c: Declare variables before code
  (write_reject_log_url):
    Use const keyword where appropriate
    Use the 'default' switch statement
    Use xfree() instead of free()
    Renamed variable f -> fp
  (write_reject_log_reason):
    Use const keyword where appropriate
    Use the 'default' switch statement
    Renamed variable f -> fp
    Renamed variable r -> reason
2015-08-07 13:42:30 +02:00
Tim Rühsen
474935665e Remove redundant definition of _GNU_SOURCE
* src/warc.c: Remove definition of _GNU_SOURCE

_GNU_SOURCE is already defined in config.h
2015-08-07 13:24:14 +02:00
Jookia
e4db00d74d Add option to write URL rejections to a tab-delimited CSV log.
* main.c: Add "--rejected-log" option.
 * init.c: Add "rejectedlog" command.
 * options.h: Add "rejected_log" parameter string.
 * wget.texi: Add brief documentation on new --rejected-log option.
 * recur.c: Optionally log details of URLs not traversed.
   Add reject_reason enum.
   (download_child_p -> download_child): Return a reject_reason.
   (descend_redirect_p -> descend_redirect): Return a reject_reason.
   (retrieve_tree): Support logging reasons for rejection.
   Add write_reject_log_header that writes a CSV format header to a file.
   Add write_reject_log_url that writes a url struct to a file in CSV format.
   Add write_reject_log_reason that writes the URL and parent URL as well as the
   rejection reason to a CSV file.
 * Test--rejected-log.px: Add a basic test for the --rejected-log command.
 * tests/Makefile.am: Run Test--rejected-log.px.

This allows you to figure out why URLs are being rejected and some context
around it. CSV is used as the output format since it can be used easily parsed,
it's delimited by tabs instead of commas to allow using all (quoted) URL
characters and includes column names which may be used for compatibility.
2015-08-06 08:10:55 +02:00
Tim Rühsen
670eb924e7 Fix memory leak in HSTS code
* src/main.c (get_hsts_database): Free 'home' variable
2015-08-04 17:41:54 +02:00
Tim Rühsen
5d55018ce6 void uninitialized variable in metalink code
* src/metalink.c: Init retr_err with METALINK_MISSING_RESOURCE
* src/wget.h: Add enum METALINK_MISSING_RESOURCE
2015-08-04 17:24:59 +02:00
Darshit Shah
4e56a91001 Fix function name collision with OpenSSL library
* src/utils.[ch], src/http.c, src/metalink.c: Rename function
    hex_to_string() to wg_hex_to_string sine it collides with a
    similarly named function in OpenSSL Library.
2015-07-24 23:52:43 +05:30
Alex Henrie
b6e242cd6f Make the filename marquee a proper marquee
* src/progress.c: Start the marquee in the middle of the available space
  and do not restart it until all of the text has scrolled out of view.
2015-07-22 16:52:20 +05:30
Ander Juaristi
b60131a399 Added support for HSTS.
* Makefile.am: Added new source files hsts.c and hsts.h.
 * http.c (parse_strict_transport_security): new function for STS header
   parsing.
   (gethttp): update the HSTS store.
 * http.h: new include "hsts.h".
 * init.c: new options --hsts and --hsts-file.
 * main.c (get_hsts_database, load_hsts, save_hsts): new functions.
   New options --no-hsts and --hsts-file added to help.
   (main): load and save HSTS store.
 * options.h: new variables for supporting --hsts and --hsts-file.
 * retr.c (retrieve_url): rewrite the URI according to the HSTS policy before
   entering http_loop.
 * test.c, test.h: new unit tests for HSTS.
 * utils.c, utils.h (countchars): new function.
 * wget.h: new preprocessor check.
 * hsts.c, hsts.h: new files with the HSTS engine implementation.

Added support for HTTP Strict Transport Security (HSTS), as defined by RFC
6797.
2015-07-20 15:55:57 +02:00
Giuseppe Scrivano
9e12b8ca39 fix compiler warnings
* src/utils.h: Include <stdlib.h>
* src/recur.c: Include "exits.h"
2015-07-20 15:37:52 +02:00
Hubert Tarasiuk
6064f21c66 Geolocation support for Metalink resources.
* doc/wget.text: Add information about --preferred-location.
* src/init.c: Add --preferred-location option.
* src/main.c (option_data): Handle --preferred-location argument.
(main): Sort resources based on location if requested.
* src/metalink.c (metalink_res_cmp): Compare based on location if
priority and preference are equal.
* src/options.h (options): Add preferred_location option.
2015-07-20 15:31:06 +02:00
Hubert Tarasiuk
97389a7497 Support at most one file signature. Adapt comments to libmetalink 0.13.
* src/metalink.c (retrieve_from_metalink): Add comment about new
libmetalink version. Do not iterate over signatures - support just one.
2015-07-20 15:31:06 +02:00
Hubert Tarasiuk
225a87d4a2 Move some Metalink-related code from http.c to metalink.c.
* src/http.c: Move find_key_value, has_key, find_key_values.
* src/metalink.c: To here.
* src/metalink.h: Make them non-static and add prototypes here.
2015-07-20 15:31:06 +02:00
Hubert Tarasiuk
92a889b278 Unit test for find_key_values.
* src/http.c: Add test_find_key_values.
* src/test.c (main): Run new test.
* src/test.h: Add test_find_key_values.
2015-07-20 15:31:06 +02:00
Hubert Tarasiuk
1113e78534 Unit test for has_key.
* src/http.c: Add test_has_key.
* src/test.c (main): Run new test.
* src/test.h: Add test_has_key.
2015-07-20 15:31:06 +02:00
Hubert Tarasiuk
70cbd59ed6 Unit test for find_key_value.
* src/http.c: Add test_find_key_value.
* src/test.c (main): Run new test.
* src/test.h: Add test_find_key_value.
2015-07-20 15:31:06 +02:00
Hubert Tarasiuk
37b58e3976 Metalink support.
* bootstrap.conf: Add crypto/sha256
* configure.ac: Look for libmetalink and GPGME
* doc/wget.texi: Add --input-metalink and --metalink-over-http
options description.
* po/POTFILES.in: Add metalink.c
* src/Makefile.am: Add new translation unit (metalink.c)
* src/http.c (http_stat): Add metalink field.
(free_stat): Free metalink field.
(find_key_value): Find value of given key in header string.
(has_key): Check if token exists in header string.
(find_key_values): Find all key=value pairs in header string.
(metalink_from_http): Obtain Metalink metadata from HTTP response.
(gethttp): Call metalink_from_http if requested.
(http_loop): Request Metalink metadata from HTTP response if should be.
Fall back to regular download if no Metalink metadata found.
* src/init.c: Add --input-metalink and --metalink-over-http options
* src/main.c (option_data): Handle --input-metalink and
--metalink-over-http cmd arguments.
(print_help): Print --input-metalink option description.
(main): Retrieve files from Metalink file
* src/metalink.c (retrieve_from_metalink): Download files described by
metalink.
(metalink_res_cmp): Comparator for resources priority-sorting.
* src/metalink.h: Create header for metalink.c
(RES_TYPE_SUPPORTED): Define supported resources media.
(DEFAULT_PRI): Default mirror priority for Metalink over HTTP.
(VALID_PRI_RANGE): Valid priority range.
* src/options.h (options): Add input_metalink option and metalink_over_http
options.
* src/utils.c (hex_to_string): Convert binary data to ASCII-hex.
* src/utils.h (hex_to_string): Add prototype.
* src/wget.h: Add metalink-related error enums
Add METALINK_METADATA flag for document type.
2015-07-20 15:30:39 +02:00
Romain Bentz
80303366ae Add NULL value check to fix #45289
* src/recur.c (retrieve_tree): Check return value of url_parse()
2015-07-15 18:10:08 +02:00
Tim Rühsen
25c9b462bf Change function params to const in src/iri.[ch]
* iri.h, iri.c: Added const attribute for params of parse_charsset(),
	check_encoding_name(), idn_encode(), idn_decode(),
	remote_to_utf8(), set_uri_encoding(), set_content_encoding().
2015-07-01 17:15:10 +02:00
Tim Rühsen
77f5a27e65 Work around a libidn <= 1.30 vulnerability
* src/iri.c: Add _utf8_is_valid() to check UTF-8 sequences before
  passing them to idna_to_ascii_8z().
2015-07-01 17:15:05 +02:00
Ángel González
ae58d8a78b Fix wgetrc filename creation for Windows
* init.c/wgetrc_file_name: Remove obsolete code in WINDOWS code path

Reported-by: Gisle Vanem <gvanem@yahoo.no>
2015-06-27 21:32:48 +02:00
Tim Rühsen
c6ac51d5bc Move test_* function protoypes from test.c to test.h
* src/test.c: Remove test_* function prototypes, make tests_run static
* src/test.h: Add test_* function protoypes
2015-06-13 22:34:36 +02:00
Hubert Tarasiuk
8a8d138dcc Support If-Modified-Since header in timestamping mode.
* src/wget.h: Add IF_MODIFIED_SINCE enum for dt. Add TIMECONV_ERR
enum to uerr_t.
* src/http.c (time_to_rfc1123): Convert time_t do http time.
* src/http.c (initialize_request): Include If-Modified-Since header
if appropriate.
* src/http.c (set_file_timestamp): Separate this code from check_file_output.
* src/http.c (check_file_output): Use set_file_timestamp.
* src/http.c (gethttp): Handle properly 304 return code and 200 if server
ignores If-Modified-Since headers.
* src/http.c (http_loop): Load filename to hstat if condget was requested,
use IF_MODIFIED_SINCE if requested and current timestamp can be obtained.
2015-05-22 11:08:30 +02:00
Hubert Tarasiuk
0e8d2d4251 Add --if-modified-since option
* src/init.c: Add to commands array.
* src/main.c: Add to cmdline_option. Add to help message.
* src/options.h: Add to options struct.
2015-05-22 11:08:30 +02:00
Ander Juaristi
b0820d553b Fixed incorrect handling of reserved chars.
* src/iri.c (do_conversion): Call url_unescape_except_reserved,
instead of url_unescape.

* src/url.c (url_unescape_1): New static function.
(url_unescape): Calls url_unescape_1 with mask zero. Preserves
same behavior as before. Only code changes.
(url_unescape_except_reserved): New function.

* src/url.h: Added prototype for url_unescape_except_reserved().

When the locale is US-ASCII, URIs that contain special characters
in them are converted to IRIs according to RFC 3987, section 3.2
"Converting URIs to IRIs".
2015-05-12 21:24:06 +02:00
Darshit Shah
b6b1388fb7 Fix documentation for update_speed_ring()
* progress.c (update_speed_ring): The comment for the function
    incorrectly stated that the function uses thirty samples from the
    past instead of twenty.

    Reported-By: Yi Li <lovelylich@gmail.com>
2015-05-07 11:29:07 +05:30
Darshit Shah
9b1dd6dab8 Remove shadowed variable in http.c
* http.c (gethttp): Rename err to conn_err to prevent shadowed
    variable
2015-05-04 21:45:26 +05:30