moparisthebest
/
wget
mirror of https://github.com/moparisthebest/wget
1
0
Fork 0
Code Issues Releases Wiki Activity

Josh's patch to remove robots.txt when appriopriate.

This commit is contained in:
Micah Cowan 2007-10-14 19:00:33 -07:00
parent 613c317785
commit 38b0f4ba27
2 changed files with 14 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

5
src/ChangeLog
View File

@ -1,3 +1,8 @@
2007-10-14 Joshua David Williams <yurimxpxman@gmail.com>
* recur.c (download_child_p): Remove robots.txt if
--delete-after or --spider is on.
2007-10-13 Micah Cowan <micah@cowan.name>
* Makefile.am: Make version.c depend on Wget dependencies (source

9
src/recur.c
View File

@ -570,6 +570,15 @@ download_child_p (const struct urlpos *upos, struct url *parent, int depth,
if (res_retrieve_file (url, &rfile))
{
specs = res_parse_from_file (rfile);
/* Delete the robots.txt file if we chose to either delete the
files after downloading or we're just running a spider. */
if (opt.delete_after || opt.spider)
{
logprintf (LOG_VERBOSE, "Removing %s.\n", rfile);
unlink (rfile);
}
xfree (rfile);
}
else