Commit Graph

362 Commits

Author SHA1 Message Date
hniksic 562ab4ae8a [svn] (DO_REALLOC_FROM_ALLOCA): Multiply with sizeof(type) when calling
xmalloc and memcpy.
Submitted by Andreas Damm in
<Pine.LNX.4.33.0202011632280.1281-200000@gate.rad-ab.u-net.com>.
2002-02-18 21:32:59 -08:00
hniksic 667d494978 [svn] Only include Unix-specific includes #ifndef WINDOWS. 2002-02-18 21:25:42 -08:00
hniksic 75a080ad0d [svn] Follow https links from http.
Submitted by Christian Lackas in <20020211202444.GA20371@lackas.desy.de>.
2002-02-18 21:23:35 -08:00
hniksic 57f36f8a6a [svn] Add the necessary includes for compilation under FreeBSD.
Submitted by Alan Eldridge in <200202130630.g1D6U3S07907@wwweasel.geeksrus.net>.
2002-02-18 21:21:42 -08:00
hniksic a9ff5853a9 [svn] when -c used, mark already fully retrieved file as successfully retrieved.
Submitted by Ian Abbott in <3C6019F4.30803.FF43E9@localhost>.
2002-02-18 21:18:43 -08:00
hniksic b9e90c34b4 [svn] Don't treat '?' as a query string separator when parsing FTP URLs.
Published in <sxslmdqxdmq.fsf@florida.arsdigita.de>.
2002-02-18 21:09:14 -08:00
hniksic 0e40fc9a3c [svn] Don't crash on <meta http-equiv=refresh>.
Published in <sxshep1q3je.fsf@florida.arsdigita.de>.
2002-01-31 19:34:31 -08:00
hniksic a861f718ed [svn] Enable compilation on Windows.
Submitted by Herold Heiko in
<B11A7CEC1675D511A3890002A551AD7C37D8AE@srvnt08.previnet.it>.
2002-01-30 18:23:04 -08:00
hniksic 6c4c785cc2 [svn] Remove redirection cycle detection.
Published in <sxs7kpzhcwa.fsf@florida.arsdigita.de>.
2002-01-30 11:12:20 -08:00
hniksic f2bb99d102 [svn] Wrap numeric IPv6 addresses in square brackets when sending out HTTP
`Host' header.
Published in <sxs7kq4vgzf.fsf@florida.arsdigita.de>.
2002-01-26 15:19:10 -08:00
hniksic 1bea726393 [svn] Allow all hex digits in IPv6 IP addresses.
Published in <sxsofjgvo72.fsf@florida.arsdigita.de>.
2002-01-26 12:43:17 -08:00
hniksic ef1eda86c4 [svn] Allow IPv6 numeric addresses in URLs.
Submitted in <sxsu1t9uedf.fsf@florida.arsdigita.de>.
2002-01-26 11:00:38 -08:00
hniksic a69cb0c3ad [svn] Various IPv6 fixes.
Submitted in <sxsd6zz2jf7.fsf@florida.arsdigita.de>.
2002-01-24 19:34:23 -08:00
hniksic 853a1c9e35 [svn] Fix some IPv6-related problems.
Submitted in <sxsg04w6zl6.fsf@florida.arsdigita.de>.
2002-01-23 22:19:35 -08:00
hniksic cb40034035 [svn] Commit IPv6 support by Thomas Lussnig.
Published in <sxsk7u8715e.fsf@florida.arsdigita.de>.
2002-01-23 21:45:54 -08:00
hniksic e59a7ee8ac [svn] Make -P work on Windows.
Submitted by Ian Abbott in <3C447E8F.13424.16ED42@localhost>.
2002-01-16 17:03:33 -08:00
hniksic fb2ed23830 [svn] Skip initial whitespace before testing for EOL.
Submitted in <sxsu1tl6bf2.fsf@florida.arsdigita.de>.
2002-01-16 16:59:13 -08:00
hniksic 31d20e644c [svn] select the file descriptor only if no data is pending in SSL buffers.
Submitted by tony@bluetail.com.
Published in <sxslmezocyp.fsf@florida.arsdigita.de>.
2002-01-15 07:24:19 -08:00
hniksic 08c935bef2 [svn] Strip trailing whitespace in header_process().
Published in <sxsadvh0xpz.fsf@florida.arsdigita.de>.
2002-01-14 07:17:34 -08:00
hniksic 46617228fa [svn] URL-decode user and password in URL.
Published in <sxsita52hg6.fsf@florida.arsdigita.de>.
2002-01-14 05:26:16 -08:00
hniksic 524a1f54dc [svn] Handle links to relative "net locations," e.g. <a href="//www.server.com/">. 2002-01-13 17:56:40 -08:00
hniksic cafe798fec [svn] Invalidate socket if get_contents encountered an error.
Published in <sxsita54sqy.fsf@florida.arsdigita.de>.
2002-01-13 17:39:16 -08:00
hniksic 11745f55bf [svn] Bump version to 1.8.1. 2001-12-24 13:02:25 -08:00
hniksic 5c859be2d4 [svn] Bump version to 1.8.1-pre3. 2001-12-19 06:59:01 -08:00
hniksic 8db1264218 [svn] Enqueue start_url in the canonical form.
Published in <sxsofkvi8zx.fsf@florida.arsdigita.de>.
2001-12-19 06:27:29 -08:00
hniksic 001cac91a4 [svn] Check for requested verbosity in logputs.
Published in <sxswuzjk0v6.fsf@florida.arsdigita.de>.
2001-12-19 01:36:58 -08:00
hniksic ca9319aaba [svn] Treat the "shortcut icon" link as inline.
Published in <sxsadwgko2q.fsf@florida.arsdigita.de>.
2001-12-18 17:15:34 -08:00
hniksic 2cf87bea8b [svn] Fix crash introduced by previous patch. 2001-12-18 14:20:14 -08:00
hniksic 40fd876c57 [svn] Descend into HTML files we've already downloaded. 2001-12-18 14:14:31 -08:00
hniksic 416671063a [svn] Propagate referrer information from retrieve_tree to retrieve_url.
Submitted by Ian Abbott in <3C1F4BFE.17436.D2D7B2@localhost>.
2001-12-18 07:22:03 -08:00
hniksic f03f033bf3 [svn] Bump version to 1.8.1-pre2. 2001-12-17 07:24:14 -08:00
hniksic eb0f9de05d [svn] Simplify indentation in sleep_between_retrievals. 2001-12-17 07:00:25 -08:00
hniksic 75699d6213 [svn] Use the new function `random_number' that doesn't depend on RAND_MAX
being defined.
Published in <sxsheqqq6xb.fsf@florida.arsdigita.de>.
2001-12-17 06:05:08 -08:00
hniksic a3500d32d7 [svn] Move path_simplify to url.c. 2001-12-14 07:46:00 -08:00
hniksic 0f120eee66 [svn] Fix error message when $WGETRC not found.
Published in <sxswuzrylnm.fsf@florida.arsdigita.de>.
2001-12-13 11:19:03 -08:00
hniksic f031900662 [svn] Don't abort when one URL references more than one file.
Published in <sxs1yhz0w1m.fsf@florida.arsdigita.de>.
2001-12-13 11:18:31 -08:00
hniksic 5b575eef31 [svn] Check for conn->ssl before using ssl_iwrite. 2001-12-13 08:46:56 -08:00
hniksic 6821132a4d [svn] Bump version to 1.8.1-pre1. 2001-12-12 23:40:50 -08:00
hniksic 3b44ca73ab [svn] Fix the broken URLs that pointed to info.webcrawler.com to point to
the new www.robotstxt.org site.
2001-12-12 23:29:05 -08:00
hniksic 3ddcea34a4 [svn] Mark <embed href=...> as external. 2001-12-12 23:18:59 -08:00
hniksic b9f370004d [svn] Cosmetic changes to get_urls_html. 2001-12-12 11:06:10 -08:00
hniksic d425985c37 [svn] Minor doc update. 2001-12-12 09:01:26 -08:00
hniksic 8817f4c1a4 [svn] Break up collect_tags_mapper into several functions.
Published in <sxsg06gla1h.fsf@florida.arsdigita.de>.
2001-12-12 07:43:01 -08:00
hniksic 1dee527a56 [svn] Use a typedef for IPv4 address.
Published in <sxsn10qrwbj.fsf@florida.arsdigita.de>.
2001-12-11 00:32:58 -08:00
hniksic b8789d7798 [svn] Fix obvious memory leaks in the VMS directory parser.
Published in <sxsadwqpbo1.fsf@florida.arsdigita.de>.
2001-12-10 21:29:52 -08:00
hniksic 1365950c01 [svn] Logging system bugfixes and improvements.
Published in <sxsofl7vdyo.fsf@florida.arsdigita.de>.
2001-12-09 21:31:45 -08:00
hniksic fb9368e889 [svn] Restore sanity check in address_list_set_faulty. 2001-12-09 18:33:08 -08:00
hniksic 943f657aa7 [svn] Rename long_to_string to number_to_string, and make it return a useful
value.
2001-12-09 18:29:12 -08:00
hniksic 997a87548c [svn] Minor fix for path_simplify.
Submitted in <sxsheqzx1qv.fsf@florida.arsdigita.de>.
2001-12-09 18:12:17 -08:00
hniksic 0e72aa6e13 [svn] Rewrite path_simplify.
Published in <sxspu5nx2yi.fsf@florida.arsdigita.de>.
2001-12-09 17:58:37 -08:00