moparisthebest/curl
1
0
Fork 0
mirror of https://github.com/moparisthebest/curl synced 2024-08-13 17:03:50 -04:00
Code Issues Releases Wiki Activity

TheArtOfHttpScripting: extended

Browse Source 
Extended the intial HTTP protcol part and added a mention of --trace and
--trace-ascii.

Replaced most URLs in the text to use example.com instead of all the
made up strange names.

Shortened a bunch of lines.
This commit is contained in:
Daniel Stenberg 2011-01-19 13:06:36 +01:00
parent b35622f387
commit b0940753c6
1 changed files with 55 additions and 39 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

94
docs/TheArtOfHttpScripting
View File

@ -1,5 +1,5 @@
Online: http://curl.haxx.se/docs/httpscripting.html Online: http://curl.haxx.se/docs/httpscripting.html
Date: May 28, 2008 Date: Jan 19, 2011
The Art Of Scripting HTTP Requests Using Curl The Art Of Scripting HTTP Requests Using Curl
============================================= =============================================
@ -38,10 +38,26 @@ Date: May 28, 2008
request a particular action, and then the server replies a few text lines request a particular action, and then the server replies a few text lines
before the actual requested content is sent to the client. before the actual requested content is sent to the client.
Using curl's option --verbose (-v as a short option) will display what kind of The client, curl, sends a HTTP request. The request contains a method (like
commands curl sends to the server, as well as a few other informational texts. GET, POST, HEAD etc), a number of request headers and sometimes a request
--verbose is the single most useful option when it comes to debug or even body. The HTTP server responds with a status line (indicating if things went
understand the curl<->server interaction. well), response headers and most often also a response body. The "body" part
is the plain data you requested, like the actual HTML or the image etc.
1.1 See the Protocol
Using curl's option --verbose (-v as a short option) will display what kind
of commands curl sends to the server, as well as a few other informational
texts.
--verbose is the single most useful option when it comes to debug or even
understand the curl<->server interaction.
Sometimes even --verbose is not enough. Then --trace and --trace-ascii offer
even more details as they show EVERYTHING curl sends and receives. Use it
like this:
curl --trace-ascii debugdump.txt http://www.example.com/
2. URL 2. URL
@ -61,10 +77,10 @@ Date: May 28, 2008
you get a web page returned in your terminal window. The entire HTML document you get a web page returned in your terminal window. The entire HTML document
that that URL holds. that that URL holds.
All HTTP replies contain a set of headers that are normally hidden, use All HTTP replies contain a set of response headers that are normally hidden,
curl's --include (-i) option to display them as well as the rest of the use curl's --include (-i) option to display them as well as the rest of the
document. You can also ask the remote server for ONLY the headers by using the document. You can also ask the remote server for ONLY the headers by using
--head (-I) option (which will make curl issue a HEAD request). the --head (-I) option (which will make curl issue a HEAD request).
4. Forms 4. Forms
@ -127,7 +143,8 @@ Date: May 28, 2008
And to use curl to post this form with the same data filled in as before, we And to use curl to post this form with the same data filled in as before, we
could do it like: could do it like:
curl --data "birthyear=1905&press=%20OK%20" http://www.hotmail.com/when/junk.cgi curl --data "birthyear=1905&press=%20OK%20" \
http://www.example.com/when.cgi
This kind of POST will use the Content-Type This kind of POST will use the Content-Type
application/x-www-form-urlencoded and is the most widely used POST kind. application/x-www-form-urlencoded and is the most widely used POST kind.
@ -204,7 +221,7 @@ Date: May 28, 2008
Put a file to a HTTP server with curl: Put a file to a HTTP server with curl:
curl --upload-file uploadfile http://www.uploadhttp.com/receive.cgi curl --upload-file uploadfile http://www.example.com/receive.cgi
6. HTTP Authentication 6. HTTP Authentication
@ -217,7 +234,7 @@ Date: May 28, 2008
To tell curl to use a user and password for authentication: To tell curl to use a user and password for authentication:
curl --user name:password http://www.secrets.com curl --user name:password http://www.example.com
The site might require a different authentication method (check the headers The site might require a different authentication method (check the headers
returned by the server), and then --ntlm, --digest, --negotiate or even returned by the server), and then --ntlm, --digest, --negotiate or even
@ -257,7 +274,7 @@ Date: May 28, 2008
Use curl to set the referer field with: Use curl to set the referer field with:
curl --referer http://curl.haxx.se http://daniel.haxx.se curl --referer http://www.example.come http://www.example.com
8. User Agent 8. User Agent
@ -273,13 +290,13 @@ Date: May 28, 2008
is time to set the User Agent field to fool the server into thinking you're is time to set the User Agent field to fool the server into thinking you're
one of those browsers. one of those browsers.
To make curl look like Internet Explorer on a Windows 2000 box: To make curl look like Internet Explorer 5 on a Windows 2000 box:
curl --user-agent "Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0)" [URL] curl --user-agent "Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0)" [URL]
Or why not look like you're using Netscape 4.73 on a Linux (PIII) box: Or why not look like you're using Netscape 4.73 on an old Linux box:
curl --user-agent "Mozilla/4.73 [en] (X11; U; Linux 2.2.15 i686)" [URL] curl --user-agent "Mozilla/4.73 [en] (X11; U; Linux 2.2.15 i686)" [URL]
9. Redirects 9. Redirects
@ -294,7 +311,7 @@ Date: May 28, 2008
To tell curl to follow a Location: To tell curl to follow a Location:
curl --location http://www.sitethatredirects.com curl --location http://www.example.com
If you use curl to POST to a site that immediately redirects you to another If you use curl to POST to a site that immediately redirects you to another
page, you can safely use --location (-L) and --data/--form together. Curl will page, you can safely use --location (-L) and --data/--form together. Curl will
@ -321,13 +338,13 @@ Date: May 28, 2008
The simplest way to send a few cookies to the server when getting a page with The simplest way to send a few cookies to the server when getting a page with
curl is to add them on the command line like: curl is to add them on the command line like:
curl --cookie "name=Daniel" http://www.cookiesite.com curl --cookie "name=Daniel" http://www.example.com
Cookies are sent as common HTTP headers. This is practical as it allows curl Cookies are sent as common HTTP headers. This is practical as it allows curl
to record cookies simply by recording headers. Record cookies with curl by to record cookies simply by recording headers. Record cookies with curl by
using the --dump-header (-D) option like: using the --dump-header (-D) option like:
curl --dump-header headers_and_cookies http://www.cookiesite.com curl --dump-header headers_and_cookies http://www.example.com
(Take note that the --cookie-jar option described below is a better way to (Take note that the --cookie-jar option described below is a better way to
store cookies.) store cookies.)
@ -338,24 +355,25 @@ Date: May 28, 2008
believing you had a previous connection). To use previously stored cookies, believing you had a previous connection). To use previously stored cookies,
you run curl like: you run curl like:
curl --cookie stored_cookies_in_file http://www.cookiesite.com curl --cookie stored_cookies_in_file http://www.example.com
Curl's "cookie engine" gets enabled when you use the --cookie option. If you Curl's "cookie engine" gets enabled when you use the --cookie option. If you
only want curl to understand received cookies, use --cookie with a file that only want curl to understand received cookies, use --cookie with a file that
doesn't exist. Example, if you want to let curl understand cookies from a page doesn't exist. Example, if you want to let curl understand cookies from a
and follow a location (and thus possibly send back cookies it received), you page and follow a location (and thus possibly send back cookies it received),
can invoke it like: you can invoke it like:
curl --cookie nada --location http://www.cookiesite.com curl --cookie nada --location http://www.example.com
Curl has the ability to read and write cookie files that use the same file Curl has the ability to read and write cookie files that use the same file
format that Netscape and Mozilla do. It is a convenient way to share cookies format that Netscape and Mozilla do. It is a convenient way to share cookies
between browsers and automatic scripts. The --cookie (-b) switch automatically between browsers and automatic scripts. The --cookie (-b) switch
detects if a given file is such a cookie file and parses it, and by using the automatically detects if a given file is such a cookie file and parses it,
--cookie-jar (-c) option you'll make curl write a new cookie file at the end of and by using the --cookie-jar (-c) option you'll make curl write a new cookie
an operation: file at the end of an operation:
curl --cookie cookies.txt --cookie-jar newcookies.txt http://www.cookiesite.com curl --cookie cookies.txt --cookie-jar newcookies.txt \
http://www.example.com
11. HTTPS 11. HTTPS
@ -371,7 +389,7 @@ Date: May 28, 2008
Curl supports encrypted fetches thanks to the freely available OpenSSL Curl supports encrypted fetches thanks to the freely available OpenSSL
libraries. To get a page from a HTTPS server, simply run curl like: libraries. To get a page from a HTTPS server, simply run curl like:
curl https://that.secure.server.com curl https://secure.example.com
11.1 Certificates 11.1 Certificates
@ -382,7 +400,7 @@ Date: May 28, 2008
can be specified on the command line or if not, entered interactively when can be specified on the command line or if not, entered interactively when
curl queries for it. Use a certificate with curl on a HTTPS server like: curl queries for it. Use a certificate with curl on a HTTPS server like:
curl --cert mycert.pem https://that.secure.server.com curl --cert mycert.pem https://secure.example.com
curl also tries to verify that the server is who it claims to be, by curl also tries to verify that the server is who it claims to be, by
verifying the server's certificate against a locally stored CA cert verifying the server's certificate against a locally stored CA cert
@ -403,17 +421,18 @@ Date: May 28, 2008
For example, you can change the POST request to a PROPFIND and send the data For example, you can change the POST request to a PROPFIND and send the data
as "Content-Type: text/xml" (instead of the default Content-Type) like this: as "Content-Type: text/xml" (instead of the default Content-Type) like this:
curl --data "<xml>" --header "Content-Type: text/xml" --request PROPFIND url.com curl --data "<xml>" --header "Content-Type: text/xml" \
--request PROPFIND url.com
You can delete a default header by providing one without content. Like you You can delete a default header by providing one without content. Like you
can ruin the request by chopping off the Host: header: can ruin the request by chopping off the Host: header:
curl --header "Host:" http://mysite.com curl --header "Host:" http://www.example.com
You can add headers the same way. Your server may want a "Destination:" You can add headers the same way. Your server may want a "Destination:"
header, and you can add it: header, and you can add it:
curl --header "Destination: http://moo.com/nowhere" http://url.com curl --header "Destination: http://nowhere" http://example.com
13. Web Login 13. Web Login
@ -444,7 +463,6 @@ Date: May 28, 2008
to do a proper login POST. Remember that the contents need to be URL encoded to do a proper login POST. Remember that the contents need to be URL encoded
when sent in a normal POST. when sent in a normal POST.
14. Debug 14. Debug
Many times when you run curl on a site, you'll notice that the site doesn't Many times when you run curl on a site, you'll notice that the site doesn't
@ -480,12 +498,10 @@ Date: May 28, 2008
RFC 2616 is a must to read if you want in-depth understanding of the HTTP RFC 2616 is a must to read if you want in-depth understanding of the HTTP
protocol. protocol.
RFC 2396 explains the URL syntax. RFC 3986 explains the URL syntax.
RFC 2109 defines how cookies are supposed to work. RFC 2109 defines how cookies are supposed to work.
RFC 1867 defines the HTTP post upload format. RFC 1867 defines the HTTP post upload format.
http://www.openssl.org is the home of the OpenSSL project
http://curl.haxx.se is the home of the cURL project http://curl.haxx.se is the home of the cURL project