moparisthebest/wget
1
0
Fork 0
mirror of https://github.com/moparisthebest/wget synced 2024-07-03 16:38:41 -04:00
Code Issues Releases Wiki Activity
f8e9a64ec7
wget/testenv/ChangeLog
Darshit Shah f8e9a64ec7 Documentation and code cleanup in test suite 
Add (lots) of documentation for various parts of the test suite in the
form of Python docstrings. Also, clean up some of the redundant code and
fix indentation issues.
2014-08-08 17:38:11 +05:30

503 lines
22 KiB
Plaintext
Raw Blame History

2014-08-08 Darshit Shah <darnir@gmail.com>
* conf/__init__.py: Add extra newline according to PEP8
* conf/{authentication,expect_header,expected_files,expected_ret_code,
files_crawled,hook_sample,local_files,reject_header,response,send_header,
server_files,urls,wget_commands}.py: Add docstrings explaining the conf file
and how it should be used
* server/http/http_server (InvalidRangeHeader): Clear TODO and eliminate
this exception. Use ServerError for all such purposes.
(_Handler): Remove reference to InvalidRangeHeader
(_handler.parse_range_header): User ServerError instead of InvalidRangeHeader
(_Handler.do_GET): Add docstring
(_Handler.do_POST): Add docstring. Also create an empty dict for rules if
no rules are supplied. Send the Location header as suggested in RFC 7231
(_Handler.do_PUT): Don't pop the server file already. Push it to later in ..
(_Handler.send_put): .. Here. If the file exists respond with a 204 No
Content message and pop the file for replacement. Do not send the
Content-Length, Content-Type headers since PUT requests should not respond
with data.
(_Handler.parse_auth_header): Fit line within 80 chars
(_Handler.check_response): Better visual indent
(_Handler.authorize_digest): Better visual indent.
(_Handler.expect_headers): Remove unused function
(_Handler.guess_type): Fix indentation
(HTTPd): Add newline according to PEP8 guidelines
(HTTPSd): Fix indentation
(StoppableHTTPServer): Add docstring
(HTTPSServer): Fix indentation
(WgetHTTPRequestHandler): Merge class into _handler.
(_Handler): Add docstring
(_Handler.parse_range_header): Fix indentation
(ServerError): Split exception into separate file ...
* exc/server_error.py: ... Here
* misc/colour_terminal.py: Add docstring, fix indentation
* test/base_test.py: Fix visual indent
* test/http_test.py: Fit within 80 char lines
2014-08-04 Darshit Shah <darnir@gmail.com>
* conf/server_conf.py: Delete file. Server configuration is now done via the
server_conf() method.
* server/http/http_server.py (StppableHTTPServer.server_sett): Delete
method required by the above hook
(HTTPd.server_sett): Same
2014-07-26 Darshit Shah <darnir@gmail.com>
* Test-*.py: Remove the '-d' switch from WGET_OPTIONS.
* test/base_test (BaseTest.gen_cmd_line): Add --debug and --no-config to the
list of switches passed to wget unconditionally.
2014-07-23 Darshit Shah <darnir@gmail.com>
* test/base_test.py (BaseTest.gen_cmd_line): Add support for running all
tests through valgrind if the relevant environment variable is set
* conf/expected_ret_code (ExpectedRetCode.__call__): Valgrind returns error
code 45 when it detects a memory leak.
* Readme: Update with details about valgrind tests
2014-07-22 Darshit Shah <darnir@gmail.com>
* (README): Remove old TODO and document SERVER_WAIT variable
2014-06-22 Darshit Shah <darnir@gmail.com>
* (conf.files_crawled): diff is a set object and needs explicit str
conversion.
2014-03-13 Zihang Chen <chsc4698@gmail.com>
* base_test.py:
(CommonMethods): Rename to BaseTest.
(BaseTest): Implement __init__ method where the class-wide variables are
initialized. Also variable names like `xxx_list` is renamed to its plural
form, e.g. `server_list` => `servers`.
(BaseTest.init_test_env): Remove name argument due to its unnecessarity.
(BaseTest.get_test_dir): Because the path of the test directory is needed
in multiple methods, this method is implemented.
(BaseTest.get_domain_addr): Rewrite the return statement utilizing str
formatting (which is more Pythonic).
(BaseTest.get_cmd_line): Rename to gen_cmd_line. Change the variables with
capitcal characters to lower ones. Also, the nested for loop is rewritten
to a plain loop using the zip function.
(BaseTest.__gen_local_filesys): Rename to gen_local_fs_snapshot. Move to
ExpectedFiles in conf/expected_files.py and is marked as a static
method. Refactor to a less verbose implementation.
(BaseTest._check_downloaded_files): Rename to __call__ to agree with the
invocation in test case classes. Move to ExpectedFiles in
conf/expected_files.py.
(BaseTest.get_server_rules): Refactor to a more Pythonic form utilizing
dict.items() and is marked static.
(BaseTest.stop_server): (new method) an abstract method which should stop
the currently using servers.
(BaseTest.instantiate_server_by): (new method) an abstract method which
should instantiate a server instance according to the given argument.
(BaseTest.__enter__): (new method) method which initialize the context
manager
(BaseTest.__exit__): (new method) method that finilize the context manager
and deal with the exceptions during the execution of the with statement,
subclasses can override this method for extensibility
* http_test.py:
(HTTPTest.__init__): Add call to super.__init__. Default values of
pre_hook, test_params, post_hook are set to None to avoid a subtle bug of
Python. Argument servers is renamed to protocols.
(HTTPTest.Server_setup): Move to BaseTest and rename to server_setup.
Calls to pre_hook_call, call_test, post_hook_call are removed.
(HTTPTest.hook_call, pre_hook_call, call_test, post_hook_call): Move to
BaseTest for that both HTTP test cases and FTP test cases may use these
methods.
(HTTPTest.init_HTTP_Server, init_HTTPS_Server): Merge and rename to
instantiate_server_by to implement the abstract method in BaseTest.
(HTTPTest.stop_HTTP_Server): Rename to stop_server to implement the
abstract method in BaseTest. Also, pull out the part where remaining
requests are gathered into a new method request_remaining.
(BaseTest.act_retcode): Rename to ret_code because ExpectedRetCode is
moved out from BaseTest, so the name act_retcode is actually a bit
verbose.
* conf/expected_ret_code.py:
(ExpectedRetCode.__call__): Rewrite the str into a more readable form.
* conf/files_crawled.py:
(FilesCrawled.__call__): Refactor this method into a more Pythonic form
utilizing the zip function.
* conf/local_files.py:
(LocalFiles__call__): Rewrite this method with the recommended with
statement.
* conf/server_conf.py:
(ServerConf.__call__): Rewrite this method due to BaseTest.server_list is
renamed to BaseTest.servers.
* conf/server_files.py:
(ServerFiles.__call__): Refactor the nested for loop into a plain one
utilizing the zip function.
* conf/urls.py:
(URLs): Rename url_list to urls.
* conf/wget_commands.py:
(WgetCommands): Rename command_list to commands, rename test_obj.options
to test_obj.wget_options.
* Test--https.py, Test-Proto.py, Test-Parallel-Proto.py: Argument servers
is changed to protocols due to change in the signature of
HTTPTest.__init__.
2014-03-13 Zihang Chen <chsc4698@gmail.com>
* test: (new package) package for test case classes
* WgetTest.py: Split into test/base_test.py and test/http_test.py.
* Test-*.py: Optimize the imports according to changes of WgetTest.py
2014-03-13 Zihang Chen <chsc4698@gmail.com>
* server: (new package) package for the server classes
* server.http: (new package) package for HTTP server
* server.ftp: (new package) package for FTP server
* HTTPServer.py: Move to server/http/http_server.py. Also change the
CERTFILE to '../certs/wget-cert.pem'.
* FTPServer.py: Move to server/ftp/ftp_server.py.
* WgetTest.py: Optimize import respect to the server classes.
2014-03-13 Zihang Chen <chsc4698@gmail.com>
* conf: (new package) package for rule classes and hook methods
* WgetTest.py:
(CommonMethods.Authentication): Move to conf/authentication.py.
(CommonMethods.ExpectHeader): Move to conf/expect_header.py.
(CommonMethods.RejectHeader): Move to conf/reject_header.py.
(CommonMethods.Response): Move to conf/response.py.
(CommonMethods.SendHeader): Move to conf/send_header.py.
(CommonMethods.ServerFiles): Move to conf/server_files.py.
(CommonMethods.LocalFiles): Move to conf/local_files.py.
(CommonMethods.ServerConf): Move to conf/server_conf.py.
(CommonMethods.WgetCommands): Move to conf/wget_commands.py.
(CommonMethods.Urls): Move to conf/urls.py.
(CommonMethods.ExpectedRetcode): Move to conf/expected_retcode.py.
(CommonMethods.ExpectedFiles): Move to conf/expected_files.py.
(CommonMethods.FilesCrawled): Move to conf/files_crawled.py.
(CommonMethods.__check_downloaded_files): Rename to
_check_downloaded_files, so that the method is callable from outside the
class.
(CommomMethods.get_server_rules): Modify so that it utilizes the conf
package.
(HTTPTest): Add a method hook_call(configs, name) to reduce duplications
in pre_hook_call, call_test and post_hook_call utilizing the conf package.
* conf/hook_sample.py: (new file) sample for hooks
* conf/rule_sample.py: (new file) sample for rules
* REAMDE: Update sections about customizing rules and hooks.
2014-03-13 Zihang Chen <chsc4698@gmail.com>
* exc: (new package) package for miscellaneous exceptions
* WgetTest.py: Move TestFailed to exc/test_failed.py.
2014-03-13 Zihang Chen <chsc4698@gmail.com>
* Test-Proto.py: Fix a typo (line 71: server to servers).
2014-03-13 Zihang Chen <chsc4698@gmail.com>
* WgetTest.py: Move WgetFile to package misc.
* README: Modify documentation respect to WgetFile.
* Test-*.py: Optimize imports about WgetFile.
2014-03-13 Zihang Chen <chsc4698@gmail.com>
* misc: (new package) package for miscellaneous modules
* ColourTerm.py: Move to package misc and rename to colour_terminal.py,
add print_color functions to reduce the use of string literals like
"BLUE", "RED" etc.
* WgetTest.py:
(CommonMethods.Server_setup): Change invocation to printer to print_blue.
(CommonMethods.FilesCrawled): Change invocation to printer to print_red.
(HTTPTest.__init__): Change invocations to printer to print_red and
print_green respectively.
2014-01-02 Darshit Shah <darnir@gmail.com>
* Makefile.am: Add new Test--https.py to list of tests and EXTRA_DIST.
Also replace all tabs with spaces in file for conformity.
* Test--https.py: New test to check if Wget works correctly with HTTPS
servers
* HTTPServer.py: Import new modules for use in HTTPS Servers
(HTTPSServer): New class that generates a SSL-wrapped socket for use in a
HTTPS Server.
(HTTPSd): HTTPS daemon class. Analogous to the HTTPd class
* WgetTest.py: Define global variables HTTP and HTTPS to reflect Server
types
(CommonMethods.exec_wget): Add the protocol information to the URL before
passing it to wget
(HTTPTest.__init__): Edit syntax. The servers variable now accepts a list of
servers defined by their type. E.g. HTTP, HTTPS.
(HTTPTest.Server_setup): Reflect change in type of variable servers.
However, we maintin the value of self.servers to allow most of the code to
remain unchanged.
(HTTPTest.init_HTTPS_Server): Initialize a HTTPS Server
* Test-Parallel-Proto.py: Edit to reflect slight change in Test Fiel Syntax.
* Test-Proto.py: Same
2014-01-02 Darshit Shah <darnir@gmail.com>
* WgetTest.py (CommonMentods.exec_wget): Wait for n seconds before calling
the Wget executable.
2013-12-27 Darshit Shah <darnir@gmail.com>
* WgetTest.py: Add modeline
(CommonMethods.ServerConf): New pre-test hook that sets
BaseHTTPRequestHandler class variables in all available servers
* HTTPServer.py (HTTPd.ServerConf): Call the respective method in the Server
to set the class variables
(StoppableHTTPServer.server_sett): Set the handler class variables
2013-12-26 Darshit Shah <darnir@gmail.com>
* WgetTest.py (HTTPTest.call_test): Correct the call to stop_HTTP_Server.
2013-12-25 Darshit Shah <darnir@gmail.com>
* WgetTest.py (CommonMehtods.exec_wget): Catch and handle exception if the
Wget executable is not found at src/wget
(HTTPTest.call_test): In case of error during execution, remove all existing
servers before quitting
2013-12-15 Darshit Shah <darnir@gmail.com>
* WgetTest.py (HTTPTest.HTTP_setup): Rename to Server_setup so it can be
easily reused for other non-HTTP servers.
(HTTPTest.__init__): Call Server_setup instead of HTTP_setup
(HTTPTest.Server_setup): Split into three more functions, that handle
pre-hooks, test execution and post-hooks respectively.
(HTTPTest.pre_hook_call): Set up and execute the pre-test hooks. Code split
from HTTPTest.Server_setup
(HTTPTest.call_test): Execute wget and log exit code. Code split from
HTTPTest.Server_setup
(HTTPTest.post_hook_call): Set up and execute post-test hooks. Code split
from HTTPTest.Server_setup
2013-10-14 Giuseppe Scrivano <gscrivan@redhat.com>
* Makefile.am (XFAIL_TESTS): Remove Test--spider-r.py.
2013-10-06 Giuseppe Scrivano <gscrivan@redhat.com>
* Makefile.am (EXTRA_DIST): Distribute test files.
2013-09-16 Darshit Shah <darnir@gmail.com>
* README: Update documentation
2013-09-14 Darshit Shah <darnir@gmail.com>
* HTTPServer.py (StoppableHTTPServer): Define object variable
request_headers which stores a list of requests received by the server
(StoppableHTTPServer.get_req_headers): Return the list of Request
Headers stored by the server
(_Handler.do_HEAD): Send the Request MEthod string for identification
(_Handler.do_GET): Same
(_Handler.__log_request): Log the request in Request_Headers list
(_Handler.send_head): Make a call to __log_request
* Test--spider-r.py: Add new list, Request_List, which contains all
the requests that Wget is expected to send. This will allow for
fine-grained tests on recursive downloading.
* WgetTest.py (CommonMethods.FilesCrawled): New Post-Test Hook, that
ensures that all the expected Files on the server were accessed as
expected.
(HTTPTest.stop_HTTP_server): On stopping server, asks it to respond
with list of all requests it received.
2013-09-13 Darshit Shah <darnir@gmail.com>
* Test--spider-r.py: Test retrieval in recursive spider mode.
* Makefile.am: add new file
2013-09-13 Darshit Shah <darnir@gmail.com>
* HTTPServer.py (_Handler.do_HEAD): If requested path is /, respond
with /index.html
(_Handler.do_HEAD): Smartly guess value of Content-Type Header from
file extension
(_Handler.guess_type): Use a preset list of extensions and
Content-Type strings. If the extension matches one in the list, use
that string, else default to "text/plain"
2013-09-13 Darshit Shah <darnir@gmail.com>
* WgetTest.py (CommonMethods._replace_substring): New method that will
replace a substring delimited by {{ }} characters by the value of
self.<substring> variable
(CommonMethods.WgetCommands): Use the _replace_substring () call to
replace the substrings in the the command line.
(CommonMethods.ServerFiles): Run the _replace_substring () method on
the File contents too.
2013-09-11 Darshit Shah <darnir@gmail.com>
* WgetTest.py (CommonMethods.exec_wget): Expect domain_list instead of
domain.
(CommonMethods.get_cmd_line): Same. Generate command line by
prepending to each file it's respective domain string
(CommonMethods.ServerFiles): Generate file_list and server_rules for
each Server and set the config details
(HTTPTest): New named parameter, servers which signifies number of
servers to spawn
(HTTPTest.HTTP_setup): This method now takes servers as a new
parameter. Instead of storing server and domain, we now store
server_list and domain_list. Each server must be initialized through a
loop.
(HTTPTest.stop_HTTP_server): Stop all servers in a loop.
* Test-Parallel-Proto.py: Prototype test file for multiple servers.
2013-09-10 Darshit Shah <darnir@gmail.com>
* WgetTest.py (HTTPTest.stop_HTTP_server): With the threaded servers,
we can simply use the socketserver.shutdown() method to close the
server instead of sending a QUIT command
* HTTPServer.py (StoppabelHTTPServer.serve_forever): Delete method. No
need to override this method anymore.
(WgetHTTPRequestHandler.do_QUIT): No longer required
(HTTPd): Rename self.server to self.server_inst to reduce ambiguity
when referenced from WgetTest
2013-09-08 Darshit Shah <darnir@gmail.com>
* README (File Structure): Add explanation about various variables
used consistently across all tests.
2013-09-07 Darshit Shah <darnir@gmail.com>
* HTTPServer.py: Remove bunch of old code artefacts
* WgetTest.py: Same
2013-09-07 Darshit Shah <darnir@gmail.com>
* HTTPServer.py (StoppableHTTPServer.server_conf): Change global
variable fileSys to an object variable. This is good programming
practice and required for parallel-wget support.
(StoppableHTTPServer.server_forever): Edit overridden method to remove
the global queue variable. No longer required under the new working
(WgetHTTPRequestHandler.do_QUIT): Don't push fileSys through the queue
(_Handler): Rename class __Handler to _Handler to match Python's
encapsulation rules
(_Handler.do_POST): fileSys is now an object variable of the server
(_Handler.do_PUT): Same
(_Handler.send_put): Same
(_Handler.send_head): Same
(HTTPd): New class that wraps around the server for Threading
(create_server): Make new object of HTTPd.
(spawn_server): Start the thread created through create_server
(ret_fileSys): Removed method. No longer required.
* WgetTest.py (HTTPTest.__init__): Don't explicitly set
self.act_retcode. Instead toggle tests_passed boolean to set the
correct return code.
(HTTPTest.HTTP_setup): We no longer call HTTPServer.spawn_server to
start a new instance of the server.
(HTTPTest.init_HTTP_server): We no longer call the old
create_server(), spawn_server() methods. Instead use the new HTTPd
class interface to create new instances of the server
(HTTPTest.stop_HTTP_server): Don't ask server to return fileSys.
2013-09-07 Darshit Shah <darnir@gmail.com>
* Test-Post.py: Test basic functionality for sending HTTP POST
requests using the --method command
* Makefile.am: Add new test
2013-09-06 Darshit Shah <darnir@gmail.com>
* WgetTest.py (CommonMethods.__check_downloaded_files): Print a
unified diff in case there is a mismatch in the file contents
2013-09-06 Darshit Shah <darnir@gmail.com>
* HTTPServer.py (WgetHTTPRequestHandler.test_cookies): Comment out the
old test_cookies code. This is no longer used and was causing problems
with expected cookies. The code will soon be removed anyways
* Test-cookie.py: Add new test for basic cookie functionality
* Test-cookie-401.py: Ensure cookies are saved during a 401 response
* Test-cookie-expires.py: Ensure that the Expires field is correctly
handled
* Test-cookies-domain-mismatch.py: Ensure that mismatched domains are
handled by Wget
* Makefile.am: Add the new tests
2013-09-06 Darshit Shah <darnir@gmail.com>
* README: New section on pending work. Will keep updating this to keep
track of work that remains to be done on this implementation
2013-09-05 Darshit Shah <darnir@gmail.com>
* Test-auth-with-content-disposition.py: Add test that ensures Content
Disposition works alongwith authentication
* Makefile.am: Add new test
2013-09-04 Darshit Shah <darnir@gmail.com>
* Test-c-full.py: Test Continue options
* Makefile.am: Add Test-c-full.py and Test-O
2013-09-02 Darshit Shah <darnir@gmail.com>
* Makefile.am: Add new Test
* Test-Head.py: New Test to ensure HEAD requests are handled correctly
2013-08-31 Darshit Shah <darnir@gmail.com>
* README: Explain that TEST_NAME needs to be unique
* Test-auth-no-challenge.py: Edit non-unique TEST_NAME
2013-08-31 Darshit Shah <darnir@gmail.com>
* HTTPTest.py (ServerError): Define new Exception for handling
internal control flow.
(StoppableHTTPServer.SendHeader): Simply pass. Do nothing. Adding
functionality here seems to crash for no apparent reason.
(stoppableHTTPServer.send_cust_headers): Minor optimization. No need
for extra variable.
(__Handler.Response): Handle explicit Response Code Rules
(__Handler.Authentication): Handle Authentication rules
(__Handler.handle_auth): Actual worker method for authentication
(__Handler.ExpectHeader): Ensure Expected Headers are received
(__Handler.RejectHeader): Ensure Blacklisted Headers are NOT received
(__Handler.send_HEAD): Dynamically call server rule functions based on
the self.rules list. This feature will later be added to POST/PUT, etc
2013-08-31 Darshit Shah <darnir@gmail.com>
* WgetTest.py: Remove import module defaultdict.
(CommonMethods.get_server_rules): server_rules should be a dict, not a
defaultdict (list).
* HTTPServer.py (WgetHTTPRequestHandler.get_rule_list): If rule does
not exist, return None. Not an emppty list.
(WgetHTTPRequestHandler.test_cookies): Rule variable is not a list
(__Handler.send_cust_headers): Same
(__Handler.custom_response): Same
(__Handler.is_authorized): Same
(__Handler.expect_headers): Same
(__Handler.reject_headers): Same
2013-08-31 Darshit Shah <darnir@gmail.com>
* README: (newfile) Simple help / instructions about using the Test
Environment.
* Makefile.am: (newfile) Makefile for the Test Environment. Uses the
Automake Parallel Test Harness
* WgetTest.py: (newfile) Base module that executes the Test.
* HTTPServer.py: (newfile) Contains the custom HTTP Server for the
Test Environment. Creates an instance of http.server in Python3.
* FTPServer.py: (newfile) Overrides methods from pyftpdlib for use in
the Test Environment. ** Work under progress **.
* ColourTerm.py: (newfile) A custom module to output coloured text to
the terminal. Known to work on POSIX shells.
* Test-Proto.py: (newfile) A prototype Test File. This should be
copied when writing a new Test Case.
* Test-Content-disposition-2.py: Test Content Disposition clobbering
* Test-Content-disposition.py: Test Content Disposition Headers
* Test-O.py: Test Output filename command
* Test-auth-basic-fail.py: Test returncode on auth failure
* Test-auth-basic.py: Test Basic Auth negotiation
* Test-auth-both.py: Test handling of Multiple auth providers. This
test currently fails.
* Test-auth-digest.py: Test Digest Auth Negotiation
* Test-auth-no-challenge-url.py: Ensure --auth-no-challenge is handled
when auth details are in-URL.
* Test-auth-no-challenge.py: Ensure --auth-no-challenge is honoured
* Test-auth-retcode.py: Ensure correct return code after 403 Forbidden
response.
Reference in New Issue View Git Blame Copy Permalink