moparisthebest/wget
1
0
Fork 0
mirror of https://github.com/moparisthebest/wget synced 2024-07-03 16:38:41 -04:00
Code Issues Releases Wiki Activity

Fix links to original Robots Exclusion Standard

Browse Source 
* doc/wget.texi: Fix links
This commit is contained in:
Tim Rühsen 2016-03-10 16:15:36 +01:00
parent 7cb9efa668
commit 598445ebd1
1 changed files with 2 additions and 2 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

4
doc/wget.texi
View File

@ -4195,12 +4195,12 @@ server.
Until version 1.8, Wget supported the first version of the standard,
written by Martijn Koster in 1994 and available at
@url{http://www.robotstxt.org/wc/norobots.html}. As of version 1.8,
@url{http://www.robotstxt.org/robotstxt.html}. As of version 1.8,
Wget has supported the additional directives specified in the internet
draft @samp{<draft-koster-robots-00.txt>} titled ``A Method for Web
Robots Control''. The draft, which has as far as I know never made to
an @sc{rfc}, is available at
@url{http://www.robotstxt.org/wc/norobots-rfc.txt}.
@url{http://www.robotstxt.org/norobots-rfc.txt}.
This manual no longer includes the text of the Robot Exclusion Standard.