moparisthebest/wget
1
0
Fork 0
mirror of https://github.com/moparisthebest/wget synced 2024-07-03 16:38:41 -04:00
Code Issues Releases Wiki Activity
master
wget/testenv/Test--spider-r.py

107 lines
2.4 KiB
Python
Raw Permalink Normal View History

Introducing Python based Test Environment Squashed Commit, of the following commits: 7743384 Update documentation to reflect changes in code b703633 Add feature that allows to ensure that Wget correctly crawls the website in recursive mode 0758f47 Add new test for recursive spider mode 43bb61b Smartly guess content type header d4d0e63 Support substring replace in File Contents too f578500 Compatibility fix with multiple servers 8b1a9b6 Extend Functionality to support spawning multiple servers e84192a Use the provided calls to shutdown server instead of rewriting it 99659f3 Improve Documentation cb94e52 Slight code cleanup. Remove unused code 886ac1a Shift to new Threading Model from Multiprocessing model e74c2ec Add new test for POST Requests 48644f1 Print diff when file contents don't match b6f9efe Add tests for Cookie support 4c9e6b4 Document pending work e13bc90 Add new test to ensure Content Disposition and Auth work together 60d1f4d Add new Test for Continue command 738b299 Add test, Test-Head 9b9d16b Edit non-unique TEST_NAME variable ae958db Minor optimizations to the way Server Rules are executed 50b4f0c The rules need not be a defaultdict. dccc154 Introducing Python based Test Environment
2014-07-24 06:55:44 -04:00
#!/usr/bin/env python3
from sys import exit
Refactor the Python based test suite This is a squashed commit of the following from parallel-wget: ecd6977 Refactor mainly the test cases classes d26c8eb Create package test for test case classes 507383d Move server classes to package server.protocol 195393b Create package conf where rules and hooks are put 42e482a Create package exc and move TestFailed to exc 82f44f3 Fix a typo in Test-Proto.py 31e5f33 From WgetTest.py move WgetFile to misc 422171d Create package misc, move ColourTerm.py to misc
2014-07-24 07:16:50 -04:00
from test.http_test import HTTPTest
from misc.wget_file import WgetFile
Introducing Python based Test Environment Squashed Commit, of the following commits: 7743384 Update documentation to reflect changes in code b703633 Add feature that allows to ensure that Wget correctly crawls the website in recursive mode 0758f47 Add new test for recursive spider mode 43bb61b Smartly guess content type header d4d0e63 Support substring replace in File Contents too f578500 Compatibility fix with multiple servers 8b1a9b6 Extend Functionality to support spawning multiple servers e84192a Use the provided calls to shutdown server instead of rewriting it 99659f3 Improve Documentation cb94e52 Slight code cleanup. Remove unused code 886ac1a Shift to new Threading Model from Multiprocessing model e74c2ec Add new test for POST Requests 48644f1 Print diff when file contents don't match b6f9efe Add tests for Cookie support 4c9e6b4 Document pending work e13bc90 Add new test to ensure Content Disposition and Auth work together 60d1f4d Add new Test for Continue command 738b299 Add test, Test-Head 9b9d16b Edit non-unique TEST_NAME variable ae958db Minor optimizations to the way Server Rules are executed 50b4f0c The rules need not be a defaultdict. dccc154 Introducing Python based Test Environment
2014-07-24 06:55:44 -04:00
"""
This test executed Wget in Spider mode with recursive retrieval.
"""
TEST_NAME = "Recursive Spider"
############# File Definitions ###############################################
mainpage = """
<html>
<head>
<title>Main Page</title>
</head>
<body>
<p>
Some text and a link to a <a href="http://127.0.0.1:{{port}}/secondpage.html">second page</a>.
Also, a <a href="http://127.0.0.1:{{port}}/nonexistent">broken link</a>.
</p>
</body>
</html>
"""
secondpage = """
<html>
<head>
<title>Second Page</title>
</head>
<body>
<p>
Some text and a link to a <a href="http://127.0.0.1:{{port}}/thirdpage.html">third page</a>.
Also, a <a href="http://127.0.0.1:{{port}}/nonexistent">broken link</a>.
</p>
</body>
</html>
"""
thirdpage = """
<html>
<head>
<title>Third Page</title>
</head>
<body>
<p>
Some text and a link to a <a href="http://127.0.0.1:{{port}}/dummy.txt">text file</a>.
Also, another <a href="http://127.0.0.1:{{port}}/againnonexistent">broken link</a>.
</p>
</body>
</html>
"""
dummyfile = "Don't care."
index_html = WgetFile ("index.html", mainpage)
secondpage_html = WgetFile ("secondpage.html", secondpage)
thirdpage_html = WgetFile ("thirdpage.html", thirdpage)
dummy_txt = WgetFile ("dummy.txt", dummyfile)
Request_List = [
[
"HEAD /",
"GET /",
"GET /robots.txt",
"HEAD /secondpage.html",
"GET /secondpage.html",
"HEAD /nonexistent",
"HEAD /thirdpage.html",
"GET /thirdpage.html",
"HEAD /dummy.txt",
"HEAD /againnonexistent"
]
]
Group common switches in test suite together Some command line switches are passed to Wget unconditionally. These switches should exist in a single place instead of being redundantly defined in each test file. We add the following two switches by default here: 1. --debug: This causes wget to be most verbose and output a lot of debugging information. Hence, if a test fails, the test log should provide valuable information. 2. --no-config: Users may have their own wgetrc files on the system. However, for the tests, we want Wget to run with vanilla settings. Hence, disbale loading any config files.
2014-07-26 14:15:57 -04:00
WGET_OPTIONS = "--spider -r"
Introducing Python based Test Environment Squashed Commit, of the following commits: 7743384 Update documentation to reflect changes in code b703633 Add feature that allows to ensure that Wget correctly crawls the website in recursive mode 0758f47 Add new test for recursive spider mode 43bb61b Smartly guess content type header d4d0e63 Support substring replace in File Contents too f578500 Compatibility fix with multiple servers 8b1a9b6 Extend Functionality to support spawning multiple servers e84192a Use the provided calls to shutdown server instead of rewriting it 99659f3 Improve Documentation cb94e52 Slight code cleanup. Remove unused code 886ac1a Shift to new Threading Model from Multiprocessing model e74c2ec Add new test for POST Requests 48644f1 Print diff when file contents don't match b6f9efe Add tests for Cookie support 4c9e6b4 Document pending work e13bc90 Add new test to ensure Content Disposition and Auth work together 60d1f4d Add new Test for Continue command 738b299 Add test, Test-Head 9b9d16b Edit non-unique TEST_NAME variable ae958db Minor optimizations to the way Server Rules are executed 50b4f0c The rules need not be a defaultdict. dccc154 Introducing Python based Test Environment
2014-07-24 06:55:44 -04:00
WGET_URLS = [[""]]
Files = [[index_html, secondpage_html, thirdpage_html, dummy_txt]]
ExpectedReturnCode = 8
ExpectedDownloadedFiles = []
################ Pre and Post Test Hooks #####################################
pre_test = {
"ServerFiles" : Files
}
test_options = {
"WgetCommands" : WGET_OPTIONS,
"Urls" : WGET_URLS
}
post_test = {
"ExpectedFiles" : ExpectedDownloadedFiles,
"ExpectedRetcode" : ExpectedReturnCode,
"FilesCrawled" : Request_List
}
err = HTTPTest (
name=TEST_NAME,
pre_hook=pre_test,
test_params=test_options,
post_hook=post_test
).begin ()
exit (err)
Reference in New Issue Copy Permalink