2014-10-25 Tim Ruehsen * test/base_test.py (gen_cmd_line): generate valgrind command line if requested * README: amend description of VALGRIND_TESTS * Makefile.am: set/export VALGRIND_TESTS 2014-10-01 Darshit Shah * Makefile.am: Run the tests in Python's Optimizedmode * conf/__init__.py (gen_hook): Use try..except instead of if..else * misc/color_terminal.py: System and check will not change while a test is run. Do not test for them on every invokation of printer() * server/http/http_server.py: The ssl and re modules are required by specific functions. Load them lazily (HTTPSServer.__init__): Lazy load ssl module here (_handler.parse_range_header): Lazy load re module here (_Handler.get_rule_list): get() can return a default value. Use it (_Handler.guess_type): Same (_Handler.is_authorized): Unused function artefact. Remove (_Handler.reject_headers): Unused function artefact. Remove 2014-10-08 Darshit Shah * Makefile.am: Fix EXTRA_DIST variable for make distcheck * server/http/http_server.py (HTTPServer.__init__): Fix how CERTFILE is found when running make dist / make distcheck. 2014-09-30 Tim Ruehsen * test/base_test.py: Add --track-origins=yes to valgrind testing 2014-08-08 Darshit Shah * conf/__init__.py: Add extra newline according to PEP8 * conf/{authentication,expect_header,expected_files,expected_ret_code, files_crawled,hook_sample,local_files,reject_header,response,send_header, server_files,urls,wget_commands}.py: Add docstrings explaining the conf file and how it should be used * server/http/http_server (InvalidRangeHeader): Clear TODO and eliminate this exception. Use ServerError for all such purposes. (_Handler): Remove reference to InvalidRangeHeader (_handler.parse_range_header): User ServerError instead of InvalidRangeHeader (_Handler.do_GET): Add docstring (_Handler.do_POST): Add docstring. Also create an empty dict for rules if no rules are supplied. Send the Location header as suggested in RFC 7231 (_Handler.do_PUT): Don't pop the server file already. Push it to later in .. (_Handler.send_put): .. Here. If the file exists respond with a 204 No Content message and pop the file for replacement. Do not send the Content-Length, Content-Type headers since PUT requests should not respond with data. (_Handler.parse_auth_header): Fit line within 80 chars (_Handler.check_response): Better visual indent (_Handler.authorize_digest): Better visual indent. (_Handler.expect_headers): Remove unused function (_Handler.guess_type): Fix indentation (HTTPd): Add newline according to PEP8 guidelines (HTTPSd): Fix indentation (StoppableHTTPServer): Add docstring (HTTPSServer): Fix indentation (WgetHTTPRequestHandler): Merge class into _handler. (_Handler): Add docstring (_Handler.parse_range_header): Fix indentation (ServerError): Split exception into separate file ... * exc/server_error.py: ... Here * misc/colour_terminal.py: Add docstring, fix indentation * test/base_test.py: Fix visual indent * test/http_test.py: Fit within 80 char lines 2014-08-04 Darshit Shah * conf/server_conf.py: Delete file. Server configuration is now done via the server_conf() method. * server/http/http_server.py (StppableHTTPServer.server_sett): Delete method required by the above hook (HTTPd.server_sett): Same 2014-07-26 Darshit Shah * Test-*.py: Remove the '-d' switch from WGET_OPTIONS. * test/base_test (BaseTest.gen_cmd_line): Add --debug and --no-config to the list of switches passed to wget unconditionally. 2014-07-23 Darshit Shah * test/base_test.py (BaseTest.gen_cmd_line): Add support for running all tests through valgrind if the relevant environment variable is set * conf/expected_ret_code (ExpectedRetCode.__call__): Valgrind returns error code 45 when it detects a memory leak. * Readme: Update with details about valgrind tests 2014-07-22 Darshit Shah * (README): Remove old TODO and document SERVER_WAIT variable 2014-06-22 Darshit Shah * (conf.files_crawled): diff is a set object and needs explicit str conversion. 2014-03-13 Zihang Chen * base_test.py: (CommonMethods): Rename to BaseTest. (BaseTest): Implement __init__ method where the class-wide variables are initialized. Also variable names like `xxx_list` is renamed to its plural form, e.g. `server_list` => `servers`. (BaseTest.init_test_env): Remove name argument due to its unnecessarity. (BaseTest.get_test_dir): Because the path of the test directory is needed in multiple methods, this method is implemented. (BaseTest.get_domain_addr): Rewrite the return statement utilizing str formatting (which is more Pythonic). (BaseTest.get_cmd_line): Rename to gen_cmd_line. Change the variables with capitcal characters to lower ones. Also, the nested for loop is rewritten to a plain loop using the zip function. (BaseTest.__gen_local_filesys): Rename to gen_local_fs_snapshot. Move to ExpectedFiles in conf/expected_files.py and is marked as a static method. Refactor to a less verbose implementation. (BaseTest._check_downloaded_files): Rename to __call__ to agree with the invocation in test case classes. Move to ExpectedFiles in conf/expected_files.py. (BaseTest.get_server_rules): Refactor to a more Pythonic form utilizing dict.items() and is marked static. (BaseTest.stop_server): (new method) an abstract method which should stop the currently using servers. (BaseTest.instantiate_server_by): (new method) an abstract method which should instantiate a server instance according to the given argument. (BaseTest.__enter__): (new method) method which initialize the context manager (BaseTest.__exit__): (new method) method that finilize the context manager and deal with the exceptions during the execution of the with statement, subclasses can override this method for extensibility * http_test.py: (HTTPTest.__init__): Add call to super.__init__. Default values of pre_hook, test_params, post_hook are set to None to avoid a subtle bug of Python. Argument servers is renamed to protocols. (HTTPTest.Server_setup): Move to BaseTest and rename to server_setup. Calls to pre_hook_call, call_test, post_hook_call are removed. (HTTPTest.hook_call, pre_hook_call, call_test, post_hook_call): Move to BaseTest for that both HTTP test cases and FTP test cases may use these methods. (HTTPTest.init_HTTP_Server, init_HTTPS_Server): Merge and rename to instantiate_server_by to implement the abstract method in BaseTest. (HTTPTest.stop_HTTP_Server): Rename to stop_server to implement the abstract method in BaseTest. Also, pull out the part where remaining requests are gathered into a new method request_remaining. (BaseTest.act_retcode): Rename to ret_code because ExpectedRetCode is moved out from BaseTest, so the name act_retcode is actually a bit verbose. * conf/expected_ret_code.py: (ExpectedRetCode.__call__): Rewrite the str into a more readable form. * conf/files_crawled.py: (FilesCrawled.__call__): Refactor this method into a more Pythonic form utilizing the zip function. * conf/local_files.py: (LocalFiles__call__): Rewrite this method with the recommended with statement. * conf/server_conf.py: (ServerConf.__call__): Rewrite this method due to BaseTest.server_list is renamed to BaseTest.servers. * conf/server_files.py: (ServerFiles.__call__): Refactor the nested for loop into a plain one utilizing the zip function. * conf/urls.py: (URLs): Rename url_list to urls. * conf/wget_commands.py: (WgetCommands): Rename command_list to commands, rename test_obj.options to test_obj.wget_options. * Test--https.py, Test-Proto.py, Test-Parallel-Proto.py: Argument servers is changed to protocols due to change in the signature of HTTPTest.__init__. 2014-03-13 Zihang Chen * test: (new package) package for test case classes * WgetTest.py: Split into test/base_test.py and test/http_test.py. * Test-*.py: Optimize the imports according to changes of WgetTest.py 2014-03-13 Zihang Chen * server: (new package) package for the server classes * server.http: (new package) package for HTTP server * server.ftp: (new package) package for FTP server * HTTPServer.py: Move to server/http/http_server.py. Also change the CERTFILE to '../certs/wget-cert.pem'. * FTPServer.py: Move to server/ftp/ftp_server.py. * WgetTest.py: Optimize import respect to the server classes. 2014-03-13 Zihang Chen * conf: (new package) package for rule classes and hook methods * WgetTest.py: (CommonMethods.Authentication): Move to conf/authentication.py. (CommonMethods.ExpectHeader): Move to conf/expect_header.py. (CommonMethods.RejectHeader): Move to conf/reject_header.py. (CommonMethods.Response): Move to conf/response.py. (CommonMethods.SendHeader): Move to conf/send_header.py. (CommonMethods.ServerFiles): Move to conf/server_files.py. (CommonMethods.LocalFiles): Move to conf/local_files.py. (CommonMethods.ServerConf): Move to conf/server_conf.py. (CommonMethods.WgetCommands): Move to conf/wget_commands.py. (CommonMethods.Urls): Move to conf/urls.py. (CommonMethods.ExpectedRetcode): Move to conf/expected_retcode.py. (CommonMethods.ExpectedFiles): Move to conf/expected_files.py. (CommonMethods.FilesCrawled): Move to conf/files_crawled.py. (CommonMethods.__check_downloaded_files): Rename to _check_downloaded_files, so that the method is callable from outside the class. (CommomMethods.get_server_rules): Modify so that it utilizes the conf package. (HTTPTest): Add a method hook_call(configs, name) to reduce duplications in pre_hook_call, call_test and post_hook_call utilizing the conf package. * conf/hook_sample.py: (new file) sample for hooks * conf/rule_sample.py: (new file) sample for rules * REAMDE: Update sections about customizing rules and hooks. 2014-03-13 Zihang Chen * exc: (new package) package for miscellaneous exceptions * WgetTest.py: Move TestFailed to exc/test_failed.py. 2014-03-13 Zihang Chen * Test-Proto.py: Fix a typo (line 71: server to servers). 2014-03-13 Zihang Chen * WgetTest.py: Move WgetFile to package misc. * README: Modify documentation respect to WgetFile. * Test-*.py: Optimize imports about WgetFile. 2014-03-13 Zihang Chen * misc: (new package) package for miscellaneous modules * ColourTerm.py: Move to package misc and rename to colour_terminal.py, add print_color functions to reduce the use of string literals like "BLUE", "RED" etc. * WgetTest.py: (CommonMethods.Server_setup): Change invocation to printer to print_blue. (CommonMethods.FilesCrawled): Change invocation to printer to print_red. (HTTPTest.__init__): Change invocations to printer to print_red and print_green respectively. 2014-01-02 Darshit Shah * Makefile.am: Add new Test--https.py to list of tests and EXTRA_DIST. Also replace all tabs with spaces in file for conformity. * Test--https.py: New test to check if Wget works correctly with HTTPS servers * HTTPServer.py: Import new modules for use in HTTPS Servers (HTTPSServer): New class that generates a SSL-wrapped socket for use in a HTTPS Server. (HTTPSd): HTTPS daemon class. Analogous to the HTTPd class * WgetTest.py: Define global variables HTTP and HTTPS to reflect Server types (CommonMethods.exec_wget): Add the protocol information to the URL before passing it to wget (HTTPTest.__init__): Edit syntax. The servers variable now accepts a list of servers defined by their type. E.g. HTTP, HTTPS. (HTTPTest.Server_setup): Reflect change in type of variable servers. However, we maintin the value of self.servers to allow most of the code to remain unchanged. (HTTPTest.init_HTTPS_Server): Initialize a HTTPS Server * Test-Parallel-Proto.py: Edit to reflect slight change in Test Fiel Syntax. * Test-Proto.py: Same 2014-01-02 Darshit Shah * WgetTest.py (CommonMentods.exec_wget): Wait for n seconds before calling the Wget executable. 2013-12-27 Darshit Shah * WgetTest.py: Add modeline (CommonMethods.ServerConf): New pre-test hook that sets BaseHTTPRequestHandler class variables in all available servers * HTTPServer.py (HTTPd.ServerConf): Call the respective method in the Server to set the class variables (StoppableHTTPServer.server_sett): Set the handler class variables 2013-12-26 Darshit Shah * WgetTest.py (HTTPTest.call_test): Correct the call to stop_HTTP_Server. 2013-12-25 Darshit Shah * WgetTest.py (CommonMehtods.exec_wget): Catch and handle exception if the Wget executable is not found at src/wget (HTTPTest.call_test): In case of error during execution, remove all existing servers before quitting 2013-12-15 Darshit Shah * WgetTest.py (HTTPTest.HTTP_setup): Rename to Server_setup so it can be easily reused for other non-HTTP servers. (HTTPTest.__init__): Call Server_setup instead of HTTP_setup (HTTPTest.Server_setup): Split into three more functions, that handle pre-hooks, test execution and post-hooks respectively. (HTTPTest.pre_hook_call): Set up and execute the pre-test hooks. Code split from HTTPTest.Server_setup (HTTPTest.call_test): Execute wget and log exit code. Code split from HTTPTest.Server_setup (HTTPTest.post_hook_call): Set up and execute post-test hooks. Code split from HTTPTest.Server_setup 2013-10-14 Giuseppe Scrivano * Makefile.am (XFAIL_TESTS): Remove Test--spider-r.py. 2013-10-06 Giuseppe Scrivano * Makefile.am (EXTRA_DIST): Distribute test files. 2013-09-16 Darshit Shah * README: Update documentation 2013-09-14 Darshit Shah * HTTPServer.py (StoppableHTTPServer): Define object variable request_headers which stores a list of requests received by the server (StoppableHTTPServer.get_req_headers): Return the list of Request Headers stored by the server (_Handler.do_HEAD): Send the Request MEthod string for identification (_Handler.do_GET): Same (_Handler.__log_request): Log the request in Request_Headers list (_Handler.send_head): Make a call to __log_request * Test--spider-r.py: Add new list, Request_List, which contains all the requests that Wget is expected to send. This will allow for fine-grained tests on recursive downloading. * WgetTest.py (CommonMethods.FilesCrawled): New Post-Test Hook, that ensures that all the expected Files on the server were accessed as expected. (HTTPTest.stop_HTTP_server): On stopping server, asks it to respond with list of all requests it received. 2013-09-13 Darshit Shah * Test--spider-r.py: Test retrieval in recursive spider mode. * Makefile.am: add new file 2013-09-13 Darshit Shah * HTTPServer.py (_Handler.do_HEAD): If requested path is /, respond with /index.html (_Handler.do_HEAD): Smartly guess value of Content-Type Header from file extension (_Handler.guess_type): Use a preset list of extensions and Content-Type strings. If the extension matches one in the list, use that string, else default to "text/plain" 2013-09-13 Darshit Shah * WgetTest.py (CommonMethods._replace_substring): New method that will replace a substring delimited by {{ }} characters by the value of self. variable (CommonMethods.WgetCommands): Use the _replace_substring () call to replace the substrings in the the command line. (CommonMethods.ServerFiles): Run the _replace_substring () method on the File contents too. 2013-09-11 Darshit Shah * WgetTest.py (CommonMethods.exec_wget): Expect domain_list instead of domain. (CommonMethods.get_cmd_line): Same. Generate command line by prepending to each file it's respective domain string (CommonMethods.ServerFiles): Generate file_list and server_rules for each Server and set the config details (HTTPTest): New named parameter, servers which signifies number of servers to spawn (HTTPTest.HTTP_setup): This method now takes servers as a new parameter. Instead of storing server and domain, we now store server_list and domain_list. Each server must be initialized through a loop. (HTTPTest.stop_HTTP_server): Stop all servers in a loop. * Test-Parallel-Proto.py: Prototype test file for multiple servers. 2013-09-10 Darshit Shah * WgetTest.py (HTTPTest.stop_HTTP_server): With the threaded servers, we can simply use the socketserver.shutdown() method to close the server instead of sending a QUIT command * HTTPServer.py (StoppabelHTTPServer.serve_forever): Delete method. No need to override this method anymore. (WgetHTTPRequestHandler.do_QUIT): No longer required (HTTPd): Rename self.server to self.server_inst to reduce ambiguity when referenced from WgetTest 2013-09-08 Darshit Shah * README (File Structure): Add explanation about various variables used consistently across all tests. 2013-09-07 Darshit Shah * HTTPServer.py: Remove bunch of old code artefacts * WgetTest.py: Same 2013-09-07 Darshit Shah * HTTPServer.py (StoppableHTTPServer.server_conf): Change global variable fileSys to an object variable. This is good programming practice and required for parallel-wget support. (StoppableHTTPServer.server_forever): Edit overridden method to remove the global queue variable. No longer required under the new working (WgetHTTPRequestHandler.do_QUIT): Don't push fileSys through the queue (_Handler): Rename class __Handler to _Handler to match Python's encapsulation rules (_Handler.do_POST): fileSys is now an object variable of the server (_Handler.do_PUT): Same (_Handler.send_put): Same (_Handler.send_head): Same (HTTPd): New class that wraps around the server for Threading (create_server): Make new object of HTTPd. (spawn_server): Start the thread created through create_server (ret_fileSys): Removed method. No longer required. * WgetTest.py (HTTPTest.__init__): Don't explicitly set self.act_retcode. Instead toggle tests_passed boolean to set the correct return code. (HTTPTest.HTTP_setup): We no longer call HTTPServer.spawn_server to start a new instance of the server. (HTTPTest.init_HTTP_server): We no longer call the old create_server(), spawn_server() methods. Instead use the new HTTPd class interface to create new instances of the server (HTTPTest.stop_HTTP_server): Don't ask server to return fileSys. 2013-09-07 Darshit Shah * Test-Post.py: Test basic functionality for sending HTTP POST requests using the --method command * Makefile.am: Add new test 2013-09-06 Darshit Shah * WgetTest.py (CommonMethods.__check_downloaded_files): Print a unified diff in case there is a mismatch in the file contents 2013-09-06 Darshit Shah * HTTPServer.py (WgetHTTPRequestHandler.test_cookies): Comment out the old test_cookies code. This is no longer used and was causing problems with expected cookies. The code will soon be removed anyways * Test-cookie.py: Add new test for basic cookie functionality * Test-cookie-401.py: Ensure cookies are saved during a 401 response * Test-cookie-expires.py: Ensure that the Expires field is correctly handled * Test-cookies-domain-mismatch.py: Ensure that mismatched domains are handled by Wget * Makefile.am: Add the new tests 2013-09-06 Darshit Shah * README: New section on pending work. Will keep updating this to keep track of work that remains to be done on this implementation 2013-09-05 Darshit Shah * Test-auth-with-content-disposition.py: Add test that ensures Content Disposition works alongwith authentication * Makefile.am: Add new test 2013-09-04 Darshit Shah * Test-c-full.py: Test Continue options * Makefile.am: Add Test-c-full.py and Test-O 2013-09-02 Darshit Shah * Makefile.am: Add new Test * Test-Head.py: New Test to ensure HEAD requests are handled correctly 2013-08-31 Darshit Shah * README: Explain that TEST_NAME needs to be unique * Test-auth-no-challenge.py: Edit non-unique TEST_NAME 2013-08-31 Darshit Shah * HTTPTest.py (ServerError): Define new Exception for handling internal control flow. (StoppableHTTPServer.SendHeader): Simply pass. Do nothing. Adding functionality here seems to crash for no apparent reason. (stoppableHTTPServer.send_cust_headers): Minor optimization. No need for extra variable. (__Handler.Response): Handle explicit Response Code Rules (__Handler.Authentication): Handle Authentication rules (__Handler.handle_auth): Actual worker method for authentication (__Handler.ExpectHeader): Ensure Expected Headers are received (__Handler.RejectHeader): Ensure Blacklisted Headers are NOT received (__Handler.send_HEAD): Dynamically call server rule functions based on the self.rules list. This feature will later be added to POST/PUT, etc 2013-08-31 Darshit Shah * WgetTest.py: Remove import module defaultdict. (CommonMethods.get_server_rules): server_rules should be a dict, not a defaultdict (list). * HTTPServer.py (WgetHTTPRequestHandler.get_rule_list): If rule does not exist, return None. Not an emppty list. (WgetHTTPRequestHandler.test_cookies): Rule variable is not a list (__Handler.send_cust_headers): Same (__Handler.custom_response): Same (__Handler.is_authorized): Same (__Handler.expect_headers): Same (__Handler.reject_headers): Same 2013-08-31 Darshit Shah * README: (newfile) Simple help / instructions about using the Test Environment. * Makefile.am: (newfile) Makefile for the Test Environment. Uses the Automake Parallel Test Harness * WgetTest.py: (newfile) Base module that executes the Test. * HTTPServer.py: (newfile) Contains the custom HTTP Server for the Test Environment. Creates an instance of http.server in Python3. * FTPServer.py: (newfile) Overrides methods from pyftpdlib for use in the Test Environment. ** Work under progress **. * ColourTerm.py: (newfile) A custom module to output coloured text to the terminal. Known to work on POSIX shells. * Test-Proto.py: (newfile) A prototype Test File. This should be copied when writing a new Test Case. * Test-Content-disposition-2.py: Test Content Disposition clobbering * Test-Content-disposition.py: Test Content Disposition Headers * Test-O.py: Test Output filename command * Test-auth-basic-fail.py: Test returncode on auth failure * Test-auth-basic.py: Test Basic Auth negotiation * Test-auth-both.py: Test handling of Multiple auth providers. This test currently fails. * Test-auth-digest.py: Test Digest Auth Negotiation * Test-auth-no-challenge-url.py: Ensure --auth-no-challenge is handled when auth details are in-URL. * Test-auth-no-challenge.py: Ensure --auth-no-challenge is honoured * Test-auth-retcode.py: Ensure correct return code after 403 Forbidden response.