1
0
mirror of https://github.com/moparisthebest/wget synced 2024-07-03 16:38:41 -04:00
Commit Graph

375 Commits

Author SHA1 Message Date
Ygal Blum
ad5a283528 Fix compilation when without-ssl is selected 2015-12-03 16:12:35 +01:00
Giuseppe Scrivano
2b418d1146 Prepare release 1.17
* gnulib: sync with upstream.
* NEWS: Update.
* src/main.c: Change the copyright year.
2015-11-15 15:00:55 +01:00
Tim Rühsen
71979f1643 Adjust indentation of --no-use-server-timestamps in help output
* src/main.c: Adjust indentation of --no-use-server-timestamps
2015-10-15 21:09:59 +02:00
Ander Juaristi
4ad201a7e7 Added --convert-file-only option
* src/convert.c (convert_links_in_hashtable, convert_links):
   test for CO_CONVERT_BASENAME_ONLY.
   (convert_basename): new function.
 * src/convert.h: new constant CO_CONVERT_BASENAME_ONLY.
 * src/init.c, src/main.c, src/options.h: new option "--convert-file-only".
 * doc/wget.texi: updated documentation.

 Reviewed-by: Gabriel Somlo <somlo@cmu.edu>
2015-10-13 16:17:20 +02:00
Ander Juaristi
f8901af4e0 Added support for FTPS
* doc/wget.texi: updated documentation to reflect the new FTPS functionality.
 * src/ftp-basic.c (ftp_greeting): new function to read the server's greeting.
   (ftp_login): greeting code was previously here. Moved to ftp_greeting to
   support FTPS implicit mode.
   (ftp_auth): wrapper around the AUTH TLS command.
   (ftp_ccc): wrapper around the CCC command.
   (ftp_pbsz): wrapper around the PBSZ command.
   (ftp_prot): wraooer around the PROT command.
 * src/ftp.c (get_ftp_greeting): new static function.
   (init_control_ssl_connection): new static function to start SSL/TLS on the
   control channel.
   (getftp): added hooks to support FTPS commands (RFCs 2228 and 4217).
   (ftp_loop_internal): test for new FTPS error codes.
 * src/ftp.h: new enum 'prot_level' with available FTPS protection levels +
   prototypes of previous functions. New flag for enum 'wget_ftp_fstatus' to track
   whether the data channel has some security mechanism enabled or not.
 * src/gnutls.c (struct wgnutls_transport_context): new field 'session_data'.
   (wgnutls_close): free GnuTLS session data before exiting.
   (ssl_connect_wget): save/resume SSL/TLS session.
 * src/http.c (establish_connection): refactor ssl_connect_wget call.
   (metalink_from_http): take into account SCHEME_FTPS as well.
 * src/init.c, src/main.c, src/options.h: new command line/wgetrc options.
   (main): in recursive downloads, check for SCHEME_FTPS as well.
 * src/openssl.c (struct openssl_transport_context): new field 'sess'.
   (ssl_connect_wget): save/resume SSL/TLS session.
 * src/retr.c (retrieve_url): check new scheme SCHEME_FTPS.
 * src/ssl.h (ssl_connect_wget): refactor. New parameter of type 'int *'.
 * src/url.c. src/url.h: new scheme SCHEME_FTPS.
 * src/wget.h: new FTPS error codes.
 * src/metalink.h: support FTPS scheme.
2015-09-14 10:16:44 +02:00
Hubert Tarasiuk
84b9abbf3c Do not free Metalink structure if not initialized
* src/main.c (main): Move metalink_delete to the conditional block.
2015-09-02 09:17:37 +02:00
Ander Juaristi
ab47d9fa3a Extra debug traces for HSTS.
* src/main.c (load_hsts, save_hsts): added DEBUGP() calls to signal
   reads and saves of the HSTS database file.
2015-09-01 13:50:40 +02:00
Tim Rühsen
398699c438 Fix two leaks foudn by Coverity
* src/http.c (gethttp): Do not leak 'message'.
* src/main.c (format_and_print_line): Do not leak 'line_dup'.
2015-08-29 22:35:29 +02:00
Daniele Calore
12bae50b28 Fix #40426: Allow -r -O- only if FILE is regular
* main.c: added check of "-r -O FILE" option combination
    allow only if FILE is a regular file (bug #40426)
2015-08-16 00:16:12 +05:30
Jookia
e4db00d74d Add option to write URL rejections to a tab-delimited CSV log.
* main.c: Add "--rejected-log" option.
 * init.c: Add "rejectedlog" command.
 * options.h: Add "rejected_log" parameter string.
 * wget.texi: Add brief documentation on new --rejected-log option.
 * recur.c: Optionally log details of URLs not traversed.
   Add reject_reason enum.
   (download_child_p -> download_child): Return a reject_reason.
   (descend_redirect_p -> descend_redirect): Return a reject_reason.
   (retrieve_tree): Support logging reasons for rejection.
   Add write_reject_log_header that writes a CSV format header to a file.
   Add write_reject_log_url that writes a url struct to a file in CSV format.
   Add write_reject_log_reason that writes the URL and parent URL as well as the
   rejection reason to a CSV file.
 * Test--rejected-log.px: Add a basic test for the --rejected-log command.
 * tests/Makefile.am: Run Test--rejected-log.px.

This allows you to figure out why URLs are being rejected and some context
around it. CSV is used as the output format since it can be used easily parsed,
it's delimited by tabs instead of commas to allow using all (quoted) URL
characters and includes column names which may be used for compatibility.
2015-08-06 08:10:55 +02:00
Tim Rühsen
670eb924e7 Fix memory leak in HSTS code
* src/main.c (get_hsts_database): Free 'home' variable
2015-08-04 17:41:54 +02:00
Ander Juaristi
b60131a399 Added support for HSTS.
* Makefile.am: Added new source files hsts.c and hsts.h.
 * http.c (parse_strict_transport_security): new function for STS header
   parsing.
   (gethttp): update the HSTS store.
 * http.h: new include "hsts.h".
 * init.c: new options --hsts and --hsts-file.
 * main.c (get_hsts_database, load_hsts, save_hsts): new functions.
   New options --no-hsts and --hsts-file added to help.
   (main): load and save HSTS store.
 * options.h: new variables for supporting --hsts and --hsts-file.
 * retr.c (retrieve_url): rewrite the URI according to the HSTS policy before
   entering http_loop.
 * test.c, test.h: new unit tests for HSTS.
 * utils.c, utils.h (countchars): new function.
 * wget.h: new preprocessor check.
 * hsts.c, hsts.h: new files with the HSTS engine implementation.

Added support for HTTP Strict Transport Security (HSTS), as defined by RFC
6797.
2015-07-20 15:55:57 +02:00
Hubert Tarasiuk
6064f21c66 Geolocation support for Metalink resources.
* doc/wget.text: Add information about --preferred-location.
* src/init.c: Add --preferred-location option.
* src/main.c (option_data): Handle --preferred-location argument.
(main): Sort resources based on location if requested.
* src/metalink.c (metalink_res_cmp): Compare based on location if
priority and preference are equal.
* src/options.h (options): Add preferred_location option.
2015-07-20 15:31:06 +02:00
Hubert Tarasiuk
37b58e3976 Metalink support.
* bootstrap.conf: Add crypto/sha256
* configure.ac: Look for libmetalink and GPGME
* doc/wget.texi: Add --input-metalink and --metalink-over-http
options description.
* po/POTFILES.in: Add metalink.c
* src/Makefile.am: Add new translation unit (metalink.c)
* src/http.c (http_stat): Add metalink field.
(free_stat): Free metalink field.
(find_key_value): Find value of given key in header string.
(has_key): Check if token exists in header string.
(find_key_values): Find all key=value pairs in header string.
(metalink_from_http): Obtain Metalink metadata from HTTP response.
(gethttp): Call metalink_from_http if requested.
(http_loop): Request Metalink metadata from HTTP response if should be.
Fall back to regular download if no Metalink metadata found.
* src/init.c: Add --input-metalink and --metalink-over-http options
* src/main.c (option_data): Handle --input-metalink and
--metalink-over-http cmd arguments.
(print_help): Print --input-metalink option description.
(main): Retrieve files from Metalink file
* src/metalink.c (retrieve_from_metalink): Download files described by
metalink.
(metalink_res_cmp): Comparator for resources priority-sorting.
* src/metalink.h: Create header for metalink.c
(RES_TYPE_SUPPORTED): Define supported resources media.
(DEFAULT_PRI): Default mirror priority for Metalink over HTTP.
(VALID_PRI_RANGE): Valid priority range.
* src/options.h (options): Add input_metalink option and metalink_over_http
options.
* src/utils.c (hex_to_string): Convert binary data to ASCII-hex.
* src/utils.h (hex_to_string): Add prototype.
* src/wget.h: Add metalink-related error enums
Add METALINK_METADATA flag for document type.
2015-07-20 15:30:39 +02:00
Hubert Tarasiuk
0e8d2d4251 Add --if-modified-since option
* src/init.c: Add to commands array.
* src/main.c: Add to cmdline_option. Add to help message.
* src/options.h: Add to options struct.
2015-05-22 11:08:30 +02:00
Darshit Shah
7d5a7ef9ca main.c: Fix two potential memory leaks
Reported by: Coverity bug 1188048
2015-03-14 16:48:30 +05:30
Giuseppe Scrivano
16f1fb1d1f maint: update copyright year ranges to include 2015 2015-03-09 16:32:01 +01:00
Anderson Goulart
882ed28d59 src/main.c (--no-verbose): don't show progress bar
Fixes #44431
2015-03-09 00:44:23 +05:30
Darshit Shah
e316d253fa main.c: Use assertion to test buffer size 2015-03-07 00:38:04 +05:30
Darshit Shah
9dde436dd6 main.c: Need to explicitly disallow show_progress in -q 2015-03-02 21:40:57 +05:30
Darshit Shah
8705e27e20 progress bar: Allow display on stderr alongwith -o
This commit causes the --show-progress option to print the progress bar
to stderr even when a logfile was explicitly provided on the command
line. Such a combination allows a user to log the output of Wget while
simultaneously keeping track of the download status.
2015-01-20 20:16:20 +01:00
Tim Ruehsen
f6b28575cc src/main.c, src/warc.c: Use gnulib's base_name() instead of basename()
Reported-by: Eli Zaretskii <eliz@gnu.org>
2014-12-25 12:07:42 +01:00
Tim Rühsen
cfe7589397 gnulib: Use basename() from gnulib module 'dirname'
Avoid basename incompatibilities between POSIX and GNU implementations.
Also, libgen.h isn't needed any more which increases compatibility.
2014-12-21 17:34:14 +01:00
Benno Schulenberg
ee196f6fe5 * src/main.c (print_help): Indent description continuation lines by two spaces. 2014-12-21 14:10:40 +01:00
Benno Schulenberg
0417ef8b50 * src/main.c (print_help): Don't use a possessive instead of a plural. 2014-12-21 14:10:35 +01:00
Benno Schulenberg
56c0098063 * src/main.c (print_help): Don't use the backtick as a left quote mark in the help text. 2014-12-21 14:10:31 +01:00
Benno Schulenberg
712d607f9b * src/main.c (print_help): Remove the full stops from option descriptions.
And use semicolons to separate multiple phrases in a description.
2014-12-21 14:10:25 +01:00
Benno Schulenberg
32499e717d * src/main.c (print_help): Make all option descriptions start with a lowercase letter.
Also spell URL consistently in all uppercase, and fix the grammar.
2014-12-21 14:09:51 +01:00
Tim Rühsen
6bc2620592 src/main.c: Conditionally print help for --random-file and --egd-file
--random-file is only relevant when compiled with either OpenSSL or LibreSSL.
--egd-file is only relevant when compiled with OpenSSL
2014-12-17 12:39:00 +01:00
Tim Ruehsen
4850e9c873 Replaced xfree_null() by xfree() and nullify argument after freeing. 2014-12-01 16:15:37 +01:00
Tim Ruehsen
d9325f5db5 Fix length of program_argstring in main.c 2014-11-28 22:40:19 +01:00
Tim Rühsen
9217b864d8 Make program_name and program_argstring const 2014-11-24 10:20:51 +01:00
Darshit Shah
b6de436547 Fix missing extern declaration error for build_info.pl 2014-11-22 17:26:06 +05:30
Darshit Shah
c6b750061a Add extern declaration for version.c strings 2014-11-22 17:26:06 +05:30
Tim Rühsen
3c51ad7f02 Removed form feeds from sources and NEWS 2014-11-20 16:35:34 +01:00
Tim Rühsen
7b43510fe3 Fixes possible issues with Wget running in a turkish locale 2014-11-20 10:56:21 +01:00
Tim Rühsen
1356e90a14 Trivial fixes for C89 compliancy 2014-11-20 09:56:57 +01:00
Darshit Shah
f9646a0c14 Fix potential memory leak in main.c 2014-11-19 23:39:22 +05:30
Tim Rühsen
db621341a4 Code cleanup for redirect_output_signal()
Also fixes a compiler warning for 'make check'.
2014-11-17 11:46:57 +01:00
Tim Rühsen
9823dc1989 Fix missing _Noreturn declaration specifiers in src/main.c
Reported-by: Gisle Vanem <gvanem@yahoo.no>
2014-11-11 16:34:57 +01:00
Tim Rühsen
e4a8fe84e2 Added --crl-file to load a Certificate Revocation List (CRL) file
Reported-by: Noël Köthe <noel@debian.org>
2014-11-11 15:06:51 +01:00
Tim Rühsen
148065bc00 content for commit 6092205538 2014-10-29 16:18:01 +01:00
Daniel Stenberg
3d7797c46e main.c: update the --method description
The first line of a HTTP request is not a header, it is the start-line,
which for requests is called the request-line.

See http://tools.ietf.org/html/rfc7230#section-3.1
2014-07-23 18:46:21 +05:30
Darshit Shah
08cedb953b Mark non returning functions 2014-06-16 18:25:44 +05:30
Giuseppe Scrivano
087e17be1c Do not use exit() with a magic number 2014-06-12 18:48:48 +02:00
Giuseppe Scrivano
ee5b28367c main: make program_name not static 2014-06-10 11:37:00 +02:00
Darshit Shah
8624553a31 Whitespace and formatting changes.(Aesthetic only)
This commit makes lots of whitespace only changes. It has been ensured that this
commit does not make any changes to the functioning of the program. The only
changes that have been made are:
    * Remove trailing whitespaces
    * Convert tabs to spaces
    * Fix indentation issues in the code
    * Other aesthetic changes to the formatting of comments
2014-05-30 21:12:57 +05:30
Darshit Shah
4eeabffee6 More progress bar aesthetic changes
This commit introduces two new changes to how the progress bar looks:
1. Support the --progress=bar:noscroll option which will prevent the filename
   from scrolling in the progress bar
2. Print human readable value for the amount already downloaded for any file
2014-05-30 13:28:02 +05:30
Tim Ruehsen
38a7829dcb Fix compiler warnings 2014-05-12 12:18:50 +02:00
Darshit Shah
8c2fd06ba8 Add --show-progress to force display progress bar
This is a relatively large commit that implements two major features:

1. Implement --show-progress switch to force the display of the progress bar in
   any verbosity level
2. Edit the implementation of the progress bar so that the filename is displayed
   in the same line.
2014-05-01 01:07:43 +02:00