- use mmap() to read whole files in core instead of allocating memory
and read'ing it.
- use a new, more general, HTML parser (html-parse.c) and interface to
it from Wget (html-url.c).
- respect <meta name=robots content=nofollow> (easy with the new HTML
parser).
- use hash tables instead of linked lists in places where the lists
were used to facilitate mappings.
- rewrite the code in host.c to be more readable and faster (hash
tables instead of home-grown lists.)
- make convert_links properly convert partial URLs to complete ones
for those URLs that have *not* been downloaded.
- use HTTP persistent connections where available. very
simple-minded, caches the last connection to the server.
Published in <sxshf533d5r.fsf@florida.arsdigita.de>.
>= width of type" warning on 32-bit architectures. Got rid of it by tricking
the compiler w/ a variable.
* url.c (UNSAFE_CHAR): The macro didn't include all the illegal characters per
RFC1738, namely everything above '~'. It also generated a warning on OSes
where char =~ unsigned char. Fixed.
