mirror of
https://github.com/moparisthebest/wget
synced 2024-07-03 16:38:41 -04:00
7cbe8141d1
Squashed Commit, of the following commits: 7743384 Update documentation to reflect changes in code b703633 Add feature that allows to ensure that Wget correctly crawls the website in recursive mode 0758f47 Add new test for recursive spider mode 43bb61b Smartly guess content type header d4d0e63 Support substring replace in File Contents too f578500 Compatibility fix with multiple servers 8b1a9b6 Extend Functionality to support spawning multiple servers e84192a Use the provided calls to shutdown server instead of rewriting it 99659f3 Improve Documentation cb94e52 Slight code cleanup. Remove unused code 886ac1a Shift to new Threading Model from Multiprocessing model e74c2ec Add new test for POST Requests 48644f1 Print diff when file contents don't match b6f9efe Add tests for Cookie support 4c9e6b4 Document pending work e13bc90 Add new test to ensure Content Disposition and Auth work together 60d1f4d Add new Test for Continue command 738b299 Add test, Test-Head 9b9d16b Edit non-unique TEST_NAME variable ae958db Minor optimizations to the way Server Rules are executed 50b4f0c The rules need not be a defaultdict. dccc154 Introducing Python based Test Environment
223 lines
9.2 KiB
Plaintext
223 lines
9.2 KiB
Plaintext
2013-09-16 Darshit Shah <darnir@gmail.com>
|
|
|
|
* README: Update documentation
|
|
|
|
2013-09-14 Darshit Shah <darnir@gmail.com>
|
|
|
|
* HTTPServer.py (StoppableHTTPServer): Define object variable
|
|
request_headers which stores a list of requests received by the server
|
|
(StoppableHTTPServer.get_req_headers): Return the list of Request
|
|
Headers stored by the server
|
|
(_Handler.do_HEAD): Send the Request MEthod string for identification
|
|
(_Handler.do_GET): Same
|
|
(_Handler.__log_request): Log the request in Request_Headers list
|
|
(_Handler.send_head): Make a call to __log_request
|
|
* Test--spider-r.py: Add new list, Request_List, which contains all
|
|
the requests that Wget is expected to send. This will allow for
|
|
fine-grained tests on recursive downloading.
|
|
* WgetTest.py (CommonMethods.FilesCrawled): New Post-Test Hook, that
|
|
ensures that all the expected Files on the server were accessed as
|
|
expected.
|
|
(HTTPTest.stop_HTTP_server): On stopping server, asks it to respond
|
|
with list of all requests it received.
|
|
|
|
2013-09-13 Darshit Shah <darnir@gmail.com>
|
|
|
|
* Test--spider-r.py: Test retrieval in recursive spider mode.
|
|
* Makefile.am: add new file
|
|
|
|
2013-09-13 Darshit Shah <darnir@gmail.com>
|
|
|
|
* HTTPServer.py (_Handler.do_HEAD): If requested path is /, respond
|
|
with /index.html
|
|
(_Handler.do_HEAD): Smartly guess value of Content-Type Header from
|
|
file extension
|
|
(_Handler.guess_type): Use a preset list of extensions and
|
|
Content-Type strings. If the extension matches one in the list, use
|
|
that string, else default to "text/plain"
|
|
|
|
2013-09-13 Darshit Shah <darnir@gmail.com>
|
|
|
|
* WgetTest.py (CommonMethods._replace_substring): New method that will
|
|
replace a substring delimited by {{ }} characters by the value of
|
|
self.<substring> variable
|
|
(CommonMethods.WgetCommands): Use the _replace_substring () call to
|
|
replace the substrings in the the command line.
|
|
(CommonMethods.ServerFiles): Run the _replace_substring () method on
|
|
the File contents too.
|
|
|
|
2013-09-11 Darshit Shah <darnir@gmail.com>
|
|
|
|
* WgetTest.py (CommonMethods.exec_wget): Expect domain_list instead of
|
|
domain.
|
|
(CommonMethods.get_cmd_line): Same. Generate command line by
|
|
prepending to each file it's respective domain string
|
|
(CommonMethods.ServerFiles): Generate file_list and server_rules for
|
|
each Server and set the config details
|
|
(HTTPTest): New named parameter, servers which signifies number of
|
|
servers to spawn
|
|
(HTTPTest.HTTP_setup): This method now takes servers as a new
|
|
parameter. Instead of storing server and domain, we now store
|
|
server_list and domain_list. Each server must be initialized through a
|
|
loop.
|
|
(HTTPTest.stop_HTTP_server): Stop all servers in a loop.
|
|
* Test-Parallel-Proto.py: Prototype test file for multiple servers.
|
|
|
|
2013-09-10 Darshit Shah <darnir@gmail.com>
|
|
|
|
* WgetTest.py (HTTPTest.stop_HTTP_server): With the threaded servers,
|
|
we can simply use the socketserver.shutdown() method to close the
|
|
server instead of sending a QUIT command
|
|
* HTTPServer.py (StoppabelHTTPServer.serve_forever): Delete method. No
|
|
need to override this method anymore.
|
|
(WgetHTTPRequestHandler.do_QUIT): No longer required
|
|
(HTTPd): Rename self.server to self.server_inst to reduce ambiguity
|
|
when referenced from WgetTest
|
|
|
|
2013-09-08 Darshit Shah <darnir@gmail.com>
|
|
|
|
* README (File Structure): Add explanation about various variables
|
|
used consistently across all tests.
|
|
|
|
2013-09-07 Darshit Shah <darnir@gmail.com>
|
|
|
|
* HTTPServer.py: Remove bunch of old code artefacts
|
|
* WgetTest.py: Same
|
|
|
|
2013-09-07 Darshit Shah <darnir@gmail.com>
|
|
|
|
* HTTPServer.py (StoppableHTTPServer.server_conf): Change global
|
|
variable fileSys to an object variable. This is good programming
|
|
practice and required for parallel-wget support.
|
|
(StoppableHTTPServer.server_forever): Edit overridden method to remove
|
|
the global queue variable. No longer required under the new working
|
|
(WgetHTTPRequestHandler.do_QUIT): Don't push fileSys through the queue
|
|
(_Handler): Rename class __Handler to _Handler to match Python's
|
|
encapsulation rules
|
|
(_Handler.do_POST): fileSys is now an object variable of the server
|
|
(_Handler.do_PUT): Same
|
|
(_Handler.send_put): Same
|
|
(_Handler.send_head): Same
|
|
(HTTPd): New class that wraps around the server for Threading
|
|
(create_server): Make new object of HTTPd.
|
|
(spawn_server): Start the thread created through create_server
|
|
(ret_fileSys): Removed method. No longer required.
|
|
* WgetTest.py (HTTPTest.__init__): Don't explicitly set
|
|
self.act_retcode. Instead toggle tests_passed boolean to set the
|
|
correct return code.
|
|
(HTTPTest.HTTP_setup): We no longer call HTTPServer.spawn_server to
|
|
start a new instance of the server.
|
|
(HTTPTest.init_HTTP_server): We no longer call the old
|
|
create_server(), spawn_server() methods. Instead use the new HTTPd
|
|
class interface to create new instances of the server
|
|
(HTTPTest.stop_HTTP_server): Don't ask server to return fileSys.
|
|
|
|
2013-09-07 Darshit Shah <darnir@gmail.com>
|
|
|
|
* Test-Post.py: Test basic functionality for sending HTTP POST
|
|
requests using the --method command
|
|
* Makefile.am: Add new test
|
|
|
|
2013-09-06 Darshit Shah <darnir@gmail.com>
|
|
|
|
* WgetTest.py (CommonMethods.__check_downloaded_files): Print a
|
|
unified diff in case there is a mismatch in the file contents
|
|
|
|
2013-09-06 Darshit Shah <darnir@gmail.com>
|
|
|
|
* HTTPServer.py (WgetHTTPRequestHandler.test_cookies): Comment out the
|
|
old test_cookies code. This is no longer used and was causing problems
|
|
with expected cookies. The code will soon be removed anyways
|
|
* Test-cookie.py: Add new test for basic cookie functionality
|
|
* Test-cookie-401.py: Ensure cookies are saved during a 401 response
|
|
* Test-cookie-expires.py: Ensure that the Expires field is correctly
|
|
handled
|
|
* Test-cookies-domain-mismatch.py: Ensure that mismatched domains are
|
|
handled by Wget
|
|
* Makefile.am: Add the new tests
|
|
|
|
2013-09-06 Darshit Shah <darnir@gmail.com>
|
|
|
|
* README: New section on pending work. Will keep updating this to keep
|
|
track of work that remains to be done on this implementation
|
|
|
|
2013-09-05 Darshit Shah <darnir@gmail.com>
|
|
|
|
* Test-auth-with-content-disposition.py: Add test that ensures Content
|
|
Disposition works alongwith authentication
|
|
* Makefile.am: Add new test
|
|
|
|
2013-09-04 Darshit Shah <darnir@gmail.com>
|
|
|
|
* Test-c-full.py: Test Continue options
|
|
* Makefile.am: Add Test-c-full.py and Test-O
|
|
|
|
2013-09-02 Darshit Shah <darnir@gmail.com>
|
|
|
|
* Makefile.am: Add new Test
|
|
* Test-Head.py: New Test to ensure HEAD requests are handled correctly
|
|
|
|
2013-08-31 Darshit Shah <darnir@gmail.com>
|
|
|
|
* README: Explain that TEST_NAME needs to be unique
|
|
* Test-auth-no-challenge.py: Edit non-unique TEST_NAME
|
|
|
|
2013-08-31 Darshit Shah <darnir@gmail.com>
|
|
|
|
* HTTPTest.py (ServerError): Define new Exception for handling
|
|
internal control flow.
|
|
(StoppableHTTPServer.SendHeader): Simply pass. Do nothing. Adding
|
|
functionality here seems to crash for no apparent reason.
|
|
(stoppableHTTPServer.send_cust_headers): Minor optimization. No need
|
|
for extra variable.
|
|
(__Handler.Response): Handle explicit Response Code Rules
|
|
(__Handler.Authentication): Handle Authentication rules
|
|
(__Handler.handle_auth): Actual worker method for authentication
|
|
(__Handler.ExpectHeader): Ensure Expected Headers are received
|
|
(__Handler.RejectHeader): Ensure Blacklisted Headers are NOT received
|
|
(__Handler.send_HEAD): Dynamically call server rule functions based on
|
|
the self.rules list. This feature will later be added to POST/PUT, etc
|
|
|
|
2013-08-31 Darshit Shah <darnir@gmail.com>
|
|
|
|
* WgetTest.py: Remove import module defaultdict.
|
|
(CommonMethods.get_server_rules): server_rules should be a dict, not a
|
|
defaultdict (list).
|
|
* HTTPServer.py (WgetHTTPRequestHandler.get_rule_list): If rule does
|
|
not exist, return None. Not an emppty list.
|
|
(WgetHTTPRequestHandler.test_cookies): Rule variable is not a list
|
|
(__Handler.send_cust_headers): Same
|
|
(__Handler.custom_response): Same
|
|
(__Handler.is_authorized): Same
|
|
(__Handler.expect_headers): Same
|
|
(__Handler.reject_headers): Same
|
|
|
|
2013-08-31 Darshit Shah <darnir@gmail.com>
|
|
|
|
* README: (newfile) Simple help / instructions about using the Test
|
|
Environment.
|
|
* Makefile.am: (newfile) Makefile for the Test Environment. Uses the
|
|
Automake Parallel Test Harness
|
|
* WgetTest.py: (newfile) Base module that executes the Test.
|
|
* HTTPServer.py: (newfile) Contains the custom HTTP Server for the
|
|
Test Environment. Creates an instance of http.server in Python3.
|
|
* FTPServer.py: (newfile) Overrides methods from pyftpdlib for use in
|
|
the Test Environment. ** Work under progress **.
|
|
* ColourTerm.py: (newfile) A custom module to output coloured text to
|
|
the terminal. Known to work on POSIX shells.
|
|
* Test-Proto.py: (newfile) A prototype Test File. This should be
|
|
copied when writing a new Test Case.
|
|
* Test-Content-disposition-2.py: Test Content Disposition clobbering
|
|
* Test-Content-disposition.py: Test Content Disposition Headers
|
|
* Test-O.py: Test Output filename command
|
|
* Test-auth-basic-fail.py: Test returncode on auth failure
|
|
* Test-auth-basic.py: Test Basic Auth negotiation
|
|
* Test-auth-both.py: Test handling of Multiple auth providers. This
|
|
test currently fails.
|
|
* Test-auth-digest.py: Test Digest Auth Negotiation
|
|
* Test-auth-no-challenge-url.py: Ensure --auth-no-challenge is handled
|
|
when auth details are in-URL.
|
|
* Test-auth-no-challenge.py: Ensure --auth-no-challenge is honoured
|
|
* Test-auth-retcode.py: Ensure correct return code after 403 Forbidden
|
|
response.
|