HTTP "auth done right". See lib/README.httpauth

CHANGES

@@ -11,6 +11,16 @@ Daniel (24 November 2004)
   with Msys/Mingw on Windows.
 
 Daniel (22 November 2004)
+- Made HTTP PUT and POST requests no longer use HEAD when doing multi-pass
+  auth negotiation (NTLM, Digest and Negotiate), but instead use the request
+  keyword "properly". Details in lib/README.httpauth. This also introduces
+  CURLOPT_IOCTLFUNCTION and CURLOPT_IOCTLDATA, to be used by apps that use the
+  "any" auth alternative as then libcurl may need to send the PUT/POST data
+  more than once and thus may need to ask the app to "rewind" the read data
+  stream to start.
+
+  See also the new example using this: docs/examples/anyauthput.c
+
 - David Phillips enhanced test 518. I made it depend on a "feature" so that
   systems without getrlimit() won't attempt to test 518. configure now checks
   for getrlimit() and setrlimit() for this test case.

RELEASE-NOTES

@@ -3,13 +3,16 @@ Curl and libcurl 7.12.3
  Public curl release number:               84
  Releases counted from the very beginning: 111
  Available command line options:           100
- Available curl_easy_setopt() options:     121
+ Available curl_easy_setopt() options:     123
  Number of public functions in libcurl:    46
  Amount of public web site mirrors:        13
  Number of known libcurl bindings:         29
 
 This release includes the following changes:
 
+ o CURLOPT_IOCTLFUNCTION and CURLOPT_IOCTLDATA added. If your app uses HTTP
+   Digest, NTLM or Negotiate authentication, you will most likely want to use
+   these
  o -w time_redirect and num_redirects
  o no longer uses libcurl.def for building on Windows, OS/2 and Netware
  o builds on Windows CE
@@ -19,6 +22,7 @@ This release includes the following changes:
 
 This release includes the following bugfixes:
 
+ o HTTP PUT/POST with Digest, NTLM or Negotiate no longer uses HEAD
  o now gracefully bails out when exceeding FD_SETSIZE file descriptors
  o CURLINFO_REDIRECT_TIME works
  o building with gssapi libs and hdeaders in the default dirs

TODO-RELEASE

@@ -6,7 +6,3 @@ To get fixed in 7.12.3 (planned release: December 2004)
 47 - Peter Sylvester's patch for SRP on the TLS layer
      Awaits OpenSSL support for this, no need to support this in libcurl before
      there's an OpenSSL release that does it.
-
-51 - PUT/POST with multipass authenticaion
-     Daniel works on this.
-

docs/TODO

@@ -38,6 +38,11 @@ TODO
 
  LIBCURL - multi interface
 
+ * Add a curl_multi_fdset() alternative that returns only two arrays with file
+   desrciptors for reading and writing to allow the app to use whatever
+   function it prefers. Plus, this allows apps to avoid the FD_SETSIZE problem
+   with select().
+
  * Add curl_multi_timeout() to make libcurl's ares-functionality better.
 
  * Make sure we don't ever loop because of non-blocking sockets return

docs/curl.1

@@ -91,6 +91,11 @@ network round-trip. This is used instead of setting a specific authentication
 method, which you can do with \fI--basic\fP, \fI--digest\fP, \fI--ntlm\fP, and
 \fI--negotiate\fP. (Added in 7.10.6)
 
+Note that using --anyauth is not recommended if you do uploads from stdin,
+since it may require data to be sent twice and then the client must be able to
+rewind. If the need should arise when uploading from stdin, the upload
+operation will fail.
+
 If this option is used several times, the following occurrences make no
 difference.
 .IP "-b/--cookie <name=data>"

docs/examples/Makefile.am

@@ -9,7 +9,8 @@ EXTRA_DIST = README curlgtk.c sepheaders.c simple.c postit2.c		\
  ftpupload.c httpput.c simplessl.c ftpgetresp.c http-post.c		\
  post-callback.c multi-app.c multi-double.c multi-single.c		\
  multi-post.c fopen.c simplepost.c makefile.dj curlx.c https.c		\
- multi-debugcallback.c fileupload.c getinfo.c ftp3rdparty.c debug.c
+ multi-debugcallback.c fileupload.c getinfo.c ftp3rdparty.c debug.c	\
+ anyauthput.c
 
 all:
 	@echo "done"

docs/examples/README

@@ -1,4 +1,8 @@
-EXAMPLES
+                                  _   _ ____  _
+                              ___| | | |  _ \| |
+                             / __| | | | |_) | |
+                            | (__| |_| |  _ <| |___
+                             \___|\___/|_| \_\_____|
 
 This directory is for libcurl programming examples. They are meant to show
 some simple steps on how you can build your own application to take full
@@ -7,6 +11,8 @@ advantage of libcurl.
 If you end up with other small but still useful example sources, please mail
 them for submission in future packages and on the web site.
 
+BUILDING
+
 The Makefile.example is an example makefile that could be used to build these
 examples. Just edit the file according to your system and requirements first.
 
@@ -23,3 +29,34 @@ want you do reorganize them like:
 applications/experiments. Even if the examples in this directory use that site
 as an example URL at some places, it doesn't mean that the URLs work or that
 we expect you to actually torture our web site with your tests! Thanks.
+
+EXAMPLES
+
+anyauthput.c   - HTTP PUT using "any" authentication method
+curlgtk.c      - download using a GTK progress bar
+curlx.c        - getting file info from the remote cert data
+debug.c        - showing how to use the debug callback
+fileupload.c   - uploading to a file:// URL
+fopen.c        - fopen() layer that supports opening URLs and files
+ftp3rdparty.c  - FTP 3rd party transfer
+ftpget.c       - simple getting a file from FTP
+ftpgetresp.c   - get the response strings from the FTP server
+ftpupload.c    - upload a file to a FTP server
+getinfo.c      - get the Content-Type from the recent transfer
+getinmemory.c  - download a file to memory only
+http-post.c    - HTTP POST
+httpput.c      - HTTP PUT a local file
+https.c        - simple HTTPS transfer
+multi-app.c    - a multi-interface app
+multi-debugcallback.c - a multi-interface app using the debug callback
+multi-double.c - a multi-interface app doing two simultaneous transfers
+multi-post.c   - a multi-interface app doing a multipart formpost
+multi-single.c - a multi-interface app getting a single file
+multithread.c  - an example using multi-treading transfering multiple files
+persistant.c   - request two URLs with a persistant connection
+post-callback.c - send a HTTP POST using a callback
+postit2.c      - send a HTTP multipart formpost
+sepheaders.c   - download headers to a separate file
+simple.c       - the most simple download a URL source
+simplepost.c   - HTTP POST
+simplessl.c    - HTTPS example with certificates many options set

docs/examples/anyauthput.c

@@ -0,0 +1,135 @@
+/*****************************************************************************
+ *                                  _   _ ____  _
+ *  Project                     ___| | | |  _ \| |
+ *                             / __| | | | |_) | |
+ *                            | (__| |_| |  _ <| |___
+ *                             \___|\___/|_| \_\_____|
+ *
+ * $Id$
+ */
+
+#include <stdio.h>
+#include <fcntl.h>
+#include <sys/stat.h>
+
+#include <curl/curl.h>
+
+#if LIBCURL_VERSION_NUM < 0x070c03
+#error "upgrade your libcurl to no less than 7.12.3"
+#endif
+
+/*
+ * This example shows a HTTP PUT operation with authentiction using "any"
+ * type. It PUTs a file given as a command line argument to the URL also given
+ * on the command line.
+ *
+ * Since libcurl 7.12.3, using "any" auth and POST/PUT requires a set ioctl
+ * function.
+ *
+ * This example also uses its own read callback.
+ */
+
+/* ioctl callback function */
+static curlioerr my_ioctl(CURL *handle, curliocmd cmd, void *userp)
+{
+  int fd = (int)userp;
+
+  (void)handle; /* not used in here */
+
+  switch(cmd) {
+  case CURLIOCMD_RESTARTREAD:
+    /* mr libcurl kindly asks as to rewind the read data stream to start */
+    if(-1 == lseek(fd, 0, SEEK_SET))
+      /* couldn't rewind */
+      return CURLIOE_FAILRESTART;
+
+    break;
+
+  default: /* ignore unknown commands */
+    return CURLIOE_UNKNOWNCMD;
+  }
+  return CURLIOE_OK; /* success! */
+}
+
+/* read callback function, fread() look alike */
+size_t read_callback(void *ptr, size_t size, size_t nmemb, void *stream)
+{
+  size_t retcode;
+
+  int fd = (int)stream;
+
+  retcode = read(fd, ptr, size * nmemb);
+
+  fprintf(stderr, "*** We read %d bytes from file\n", retcode);
+
+  return retcode;
+}
+
+int main(int argc, char **argv)
+{
+  CURL *curl;
+  CURLcode res;
+  int hd ;
+  struct stat file_info;
+
+  char *file;
+  char *url;
+
+  if(argc < 3)
+    return 1;
+
+  file= argv[1];
+  url = argv[2];
+
+  /* get the file size of the local file */
+  hd = open(file, O_RDONLY) ;
+  fstat(hd, &file_info);
+
+  /* In windows, this will init the winsock stuff */
+  curl_global_init(CURL_GLOBAL_ALL);
+
+  /* get a curl handle */
+  curl = curl_easy_init();
+  if(curl) {
+    /* we want to use our own read function */
+    curl_easy_setopt(curl, CURLOPT_READFUNCTION, read_callback);
+
+    /* which file to upload */
+    curl_easy_setopt(curl, CURLOPT_READDATA, hd);
+
+    /* set the ioctl function */
+    curl_easy_setopt(curl, CURLOPT_IOCTLFUNCTION, my_ioctl);
+
+    /* pass the file descriptor to the ioctl callback as well */
+    curl_easy_setopt(curl, CURLOPT_IOCTLDATA, hd);
+
+    /* enable "uploading" (which means PUT when doing HTTP) */
+    curl_easy_setopt(curl, CURLOPT_UPLOAD, TRUE) ;
+
+    /* specify target URL, and note that this URL should also include a file
+       name, not only a directory (as you can do with GTP uploads) */
+    curl_easy_setopt(curl,CURLOPT_URL, url);
+
+    /* and give the size of the upload, this supports large file sizes
+       on systems that have general support for it */
+    curl_easy_setopt(curl, CURLOPT_INFILESIZE_LARGE, file_info.st_size);
+
+    /* tell libcurl we can use "any" auth, which lets the lib pick one, but it
+       also costs one extra round-trip and possibly sending of all the PUT
+       data twice!!! */
+    curl_easy_setopt(curl, CURLOPT_HTTPAUTH, CURLAUTH_ANY);
+
+    /* set user name and password for the authentication */
+    curl_easy_setopt(curl, CURLOPT_USERPWD, "user:password");
+
+    /* Now run off and do what you've been told! */
+    res = curl_easy_perform(curl);
+
+    /* always cleanup */
+    curl_easy_cleanup(curl);
+  }
+  close(hd); /* close the local file */
+
+  curl_global_cleanup();
+  return 0;
+}

docs/libcurl/curl_easy_setopt.3

@@ -21,7 +21,7 @@
 .\" * $Id$
 .\" **************************************************************************
 .\"
-.TH curl_easy_setopt 3 "12 Mar 2004" "libcurl 7.11.1" "libcurl Manual"
+.TH curl_easy_setopt 3 "21 Nov 2004" "libcurl 7.12.3" "libcurl Manual"
 .SH NAME
 curl_easy_setopt - set options for a curl easy handle
 .SH SYNOPSIS
@@ -142,6 +142,18 @@ don't specify a read callback, this must be a valid FILE *.
 
 This option is also known with the older name \fICURLOPT_INFILE\fP, the name
 \fICURLOPT_READDATA\fP was introduced in 7.9.7.
+.IP CURLOPT_IOCTLFUNCTION
+Function pointer that should match the \fIcurl_ioctl_callback\fP prototype
+found in \fI<curl/curl.h>\fP. This function gets called by libcurl when
+something special I/O-related needs to be done that the library can't do by
+itself. For now, rewinding the read data stream is the only action it can
+request. The rewinding of the read data stream may be necessary when doing a
+HTTP PUT or POST with a multi-pass authentication method.  (Opion added in
+7.12.3)
+.IP CURLOPT_IOCTLDATA
+Pass a pointer that will be untouched by libcurl and passed as the 3rd
+argument in the ioctl callback set with \fICURLOPT_IOCTLFUNCTION\fP.  (Option
+added in 7.12.3)
 .IP CURLOPT_PROGRESSFUNCTION
 Function pointer that should match the \fIcurl_progress_callback\fP prototype
 found in \fI<curl/curl.h>\fP. This function gets called by libcurl instead of

include/curl/curl.h

@@ -163,19 +163,39 @@ typedef size_t (*curl_write_callback)(char *buffer,
                                       size_t nitems,
                                       void *outstream);
 
-/* This is a brand new return code for the read callback that will signal
+/* This is a return code for the read callback that, when returned, will
-   the caller to immediately abort the current transfer. */
+   signal libcurl to immediately abort the current transfer. */
 #define CURL_READFUNC_ABORT 0x10000000
 typedef size_t (*curl_read_callback)(char *buffer,
                                       size_t size,
                                       size_t nitems,
                                       void *instream);
 
+
+#ifndef CURL_NO_OLDIES
   /* not used since 7.10.8, will be removed in a future release */
 typedef int (*curl_passwd_callback)(void *clientp,
                                     const char *prompt,
                                     char *buffer,
                                     int buflen);
+#endif
+
+typedef enum {
+  CURLIOE_OK,            /* I/O operation successful */
+  CURLIOE_UNKNOWNCMD,    /* command was unknown to callback */
+  CURLIOE_FAILRESTART,   /* failed to restart the read */
+  CURLIOE_LAST           /* never use */
+} curlioerr;
+
+typedef enum  {
+  CURLIOCMD_NOP,         /* no operation */
+  CURLIOCMD_RESTARTREAD, /* restart the read stream from start */
+  CURLIOCMD_LAST         /* never use */
+} curliocmd;
+
+typedef curlioerr (*curl_ioctl_callback)(CURL *handle,
+                                         int cmd,
+                                         void *clientp);
 
 /*
  * The following typedef's are signatures of malloc, free, realloc, strdup and
@@ -282,7 +302,8 @@ typedef enum {
   CURLE_LDAP_INVALID_URL,        /* 62 - Invalid LDAP URL */
   CURLE_FILESIZE_EXCEEDED,       /* 63 - Maximum file size exceeded */
   CURLE_FTP_SSL_FAILED,          /* 64 - Requested FTP SSL level failed */
-
+  CURLE_SEND_FAIL_REWIND,        /* 65 - Sending the data requires a rewind
+                                    that failed */
   CURL_LAST /* never use! */
 } CURLcode;
 
@@ -854,6 +875,9 @@ typedef enum {
   */
   CINIT(FTPSSLAUTH, LONG, 129),
 
+  CINIT(IOCTLFUNCTION, FUNCTIONPOINT, 130),
+  CINIT(IOCTLDATA, OBJECTPOINT, 131),
+
   CURLOPT_LASTENTRY /* the last unused */
 } CURLoption;
 

lib/README.httpauth

@@ -19,6 +19,11 @@
    determined. Possibly the current transfer speed should be taken into
    account as well.
 
+   NOTE: if the size of the POST data is less than MAX_INITIAL_POST_SIZE (when
+   CURLOPT_POSTFIELDS is used), libcurl will send everything in one single
+   write() (all request-headers and request-body) and thus it will
+   unconditionally send the full post data here.
+
 2. PUT/POST with multi-pass auth but not yet completely negotiated:
 
    Send a PUT/POST request, we know that it will be rejected and thus we claim

lib/http.c

@@ -192,6 +192,80 @@ static bool pickoneauth(struct auth *pick)
   return picked;
 }
 
+/*
+ * perhapsrewind()
+ *
+ * If we are doing POST or PUT {
+ *   If we have more data to send {
+ *     If we are doing NTLM {
+ *       Keep sending since we must not disconnect
+ *     }
+ *     else {
+ *       If there is more than just a little data left to send, close
+ *       the current connection by force.
+ *     }
+ *   }
+ *   If we have sent any data {
+ *     If we don't have track of all the data {
+ *       call app to tell it to rewind
+ *     }
+ *     else {
+ *       rewind internally so that the operation can restart fine
+ *     }
+ *   }
+ * }
+ */
+static CURLcode perhapsrewind(struct connectdata *conn)
+{
+  struct HTTP *http = conn->proto.http;
+  struct SessionHandle *data = conn->data;
+  curl_off_t bytessent = http->writebytecount;
+  curl_off_t expectsend = -1; /* default is unknown */
+
+  /* figure out how much data we are expected to send */
+  switch(data->set.httpreq) {
+  case HTTPREQ_POST:
+    if(data->set.postfieldsize != -1)
+      expectsend = data->set.postfieldsize;
+    break;
+  case HTTPREQ_PUT:
+    if(data->set.infilesize != -1)
+      expectsend = data->set.infilesize;
+    break;
+  case HTTPREQ_POST_FORM:
+    expectsend = http->postsize;
+    break;
+  default:
+    break;
+  }
+
+  conn->bits.rewindaftersend = FALSE; /* default */
+
+  if((expectsend == -1) || (expectsend > bytessent)) {
+    /* There is still data left to send */
+    if((data->state.authproxy.picked == CURLAUTH_NTLM) ||/* using NTLM */
+       (data->state.authhost.picked == CURLAUTH_NTLM) ) {
+      conn->bits.close = FALSE; /* don't close, keep on sending */
+
+      /* rewind data when completely done sending! */
+      conn->bits.rewindaftersend = TRUE;
+      return CURLE_OK;
+    }
+    else {
+      /* If there is more than just a little data left to send, close the
+       * current connection by force.
+       */
+      conn->bits.close = TRUE;
+      conn->size = 0; /* don't download any more than 0 bytes */
+    }
+  }
+
+  if(bytessent)
+    return Curl_readrewind(conn);
+
+  return CURLE_OK;
+}
+
 /*
  * Curl_http_auth_act() gets called when a all HTTP headers have been received
  * and it checks what authentication methods that are available and decides
@@ -211,25 +285,33 @@ CURLcode Curl_http_auth_act(struct connectdata *conn)
 
   if(conn->bits.user_passwd &&
      ((conn->keep.httpcode == 401) ||
-      (conn->bits.authprobe && conn->keep.httpcode < 300))) {
+      (conn->bits.authneg && conn->keep.httpcode < 300))) {
     pickhost = pickoneauth(&data->state.authhost);
     if(!pickhost)
       data->state.authproblem = TRUE;
   }
   if(conn->bits.proxy_user_passwd &&
      ((conn->keep.httpcode == 407) ||
-      (conn->bits.authprobe && conn->keep.httpcode < 300))) {
+      (conn->bits.authneg && conn->keep.httpcode < 300))) {
     pickproxy = pickoneauth(&data->state.authproxy);
     if(!pickproxy)
       data->state.authproblem = TRUE;
   }
 
-  if(pickhost || pickproxy)
+  if(pickhost || pickproxy) {
     conn->newurl = strdup(data->change.url); /* clone URL */
 
+    if((data->set.httpreq != HTTPREQ_GET) &&
+       (data->set.httpreq != HTTPREQ_HEAD)) {
+      code = perhapsrewind(conn);
+      if(code)
+        return code;
+    }
+  }
+
   else if((conn->keep.httpcode < 300) &&
           (!data->state.authhost.done) &&
-          conn->bits.authprobe) {
+          conn->bits.authneg) {
     /* no (known) authentication available,
        authentication is not "done" yet and
        no authentication seems to be required and
@@ -273,29 +355,34 @@ Curl_http_output_auth(struct connectdata *conn,
   CURLcode result = CURLE_OK;
   struct SessionHandle *data = conn->data;
   char *auth=NULL;
+  struct auth *authhost;
+  struct auth *authproxy;
 
   curlassert(data);
 
+  authhost = &data->state.authhost;
+  authproxy = &data->state.authproxy;
+
   if((conn->bits.httpproxy && conn->bits.proxy_user_passwd) ||
      conn->bits.user_passwd)
     /* continue please */ ;
   else {
-    data->state.authhost.done = TRUE;
+    authhost->done = TRUE;
-    data->state.authproxy.done = TRUE;
+    authproxy->done = TRUE;
     return CURLE_OK; /* no authentication with no user or password */
   }
 
-  if(data->state.authhost.want && !data->state.authhost.picked)
+  if(authhost->want && !authhost->picked)
     /* The app has selected one or more methods, but none has been picked
        so far by a server round-trip. Then we set the picked one to the
        want one, and if this is one single bit it'll be used instantly. */
-    data->state.authhost.picked = data->state.authhost.want;
+    authhost->picked = authhost->want;
 
-  if(data->state.authproxy.want && !data->state.authproxy.picked)
+  if(authproxy->want && !authproxy->picked)
     /* The app has selected one or more methods, but none has been picked so
        far by a proxy round-trip. Then we set the picked one to the want one,
        and if this is one single bit it'll be used instantly. */
-    data->state.authproxy.picked = data->state.authproxy.want;
+    authproxy->picked = authproxy->want;
 
   /* To prevent the user+password to get sent to other than the original
      host due to a location-follow, we do some weirdo checks here */
@@ -308,7 +395,7 @@ Curl_http_output_auth(struct connectdata *conn,
     if (conn->bits.httpproxy &&
         (conn->bits.tunnel_proxy == proxytunnel)) {
 #ifdef USE_SSLEAY
-      if(data->state.authproxy.want == CURLAUTH_NTLM) {
+      if(authproxy->want == CURLAUTH_NTLM) {
         auth=(char *)"NTLM";
         result = Curl_output_ntlm(conn, TRUE);
         if(result)
@@ -316,7 +403,7 @@ Curl_http_output_auth(struct connectdata *conn,
       }
       else
 #endif
-      if(data->state.authproxy.want == CURLAUTH_BASIC) {
+      if(authproxy->want == CURLAUTH_BASIC) {
         /* Basic */
         if(conn->bits.proxy_user_passwd &&
            !checkheaders(data, "Proxy-authorization:")) {
@@ -325,10 +412,12 @@ Curl_http_output_auth(struct connectdata *conn,
           if(result)
             return result;
         }
-        data->state.authproxy.done = TRUE;
+        /* NOTE: Curl_output_basic() should set 'done' TRUE, as the other auth
+           functions work that way */
+        authproxy->done = TRUE;
       }
 #ifndef CURL_DISABLE_CRYPTO_AUTH
-      else if(data->state.authproxy.want == CURLAUTH_DIGEST) {
+      else if(authproxy->want == CURLAUTH_DIGEST) {
         auth=(char *)"Digest";
         result = Curl_output_digest(conn,
                                     TRUE, /* proxy */
@@ -338,31 +427,36 @@ Curl_http_output_auth(struct connectdata *conn,
           return result;
       }
 #endif
+      if(auth) {
         infof(data, "Proxy auth using %s with user '%s'\n",
               auth, conn->proxyuser?conn->proxyuser:"");
+        authproxy->multi = !authproxy->done;
+      }
+      else
+        authproxy->multi = FALSE;
     }
     else
       /* we have no proxy so let's pretend we're done authenticating
          with it */
-      data->state.authproxy.done = TRUE;
+      authproxy->done = TRUE;
 
     /* Send web authentication header if needed */
     {
       auth = NULL;
 #ifdef HAVE_GSSAPI
-      if((data->state.authhost.want == CURLAUTH_GSSNEGOTIATE) &&
+      if((authhost->want == CURLAUTH_GSSNEGOTIATE) &&
          data->state.negotiate.context &&
          !GSS_ERROR(data->state.negotiate.status)) {
         auth=(char *)"GSS-Negotiate";
         result = Curl_output_negotiate(conn);
         if (result)
           return result;
-        data->state.authhost.done = TRUE;
+        authhost->done = TRUE;
       }
       else
 #endif
 #ifdef USE_SSLEAY
-      if(data->state.authhost.picked == CURLAUTH_NTLM) {
+      if(authhost->picked == CURLAUTH_NTLM) {
         auth=(char *)"NTLM";
         result = Curl_output_ntlm(conn, FALSE);
         if(result)
@@ -372,7 +466,7 @@ Curl_http_output_auth(struct connectdata *conn,
 #endif
       {
 #ifndef CURL_DISABLE_CRYPTO_AUTH
-        if(data->state.authhost.picked == CURLAUTH_DIGEST) {
+        if(authhost->picked == CURLAUTH_DIGEST) {
           auth=(char *)"Digest";
           result = Curl_output_digest(conn,
                                       FALSE, /* not a proxy */
@@ -382,7 +476,7 @@ Curl_http_output_auth(struct connectdata *conn,
             return result;
         } else
 #endif
-        if(data->state.authhost.picked == CURLAUTH_BASIC) {
+        if(authhost->picked == CURLAUTH_BASIC) {
           if(conn->bits.user_passwd &&
              !checkheaders(data, "Authorization:")) {
             auth=(char *)"Basic";
@@ -391,16 +485,21 @@ Curl_http_output_auth(struct connectdata *conn,
               return result;
           }
           /* basic is always ready */
-          data->state.authhost.done = TRUE;
+          authhost->done = TRUE;
         }
       }
-      if(auth)
+      if(auth) {
         infof(data, "Server auth using %s with user '%s'\n",
               auth, conn->user);
+
+        authhost->multi = !authhost->done;
+      }
+      else
+        authhost->multi = FALSE;
     }
   }
   else
-    data->state.authhost.done = TRUE;
+    authhost->done = TRUE;
 
   return result;
 }
@@ -1304,20 +1403,15 @@ CURLcode Curl_http(struct connectdata *conn)
   if(result)
     return result;
 
-  if((!data->state.authhost.done || !data->state.authproxy.done ) &&
+  if((data->state.authhost.multi || data->state.authproxy.multi) &&
-     (httpreq != HTTPREQ_GET)) {
+     (httpreq != HTTPREQ_GET) &&
-    /* Until we are authenticated, we switch over to HEAD. Unless its a GET
+     (httpreq != HTTPREQ_HEAD)) {
-       we want to do. The explanation for this is rather long and boring, but
+    /* Auth is required and we are not authenticated yet. Make a PUT or POST
-       the point is that it can't be done otherwise without risking having to
+       with content-length zero as a "probe". */
-       send the POST or PUT data multiple times. */
+    conn->bits.authneg = TRUE;
-    httpreq = HTTPREQ_HEAD;
-    request = (char *)"HEAD";
-    conn->bits.no_body = TRUE;
-    conn->bits.authprobe = TRUE; /* this is a request done to probe for
-                                    authentication methods */
   }
   else
-    conn->bits.authprobe = FALSE;
+    conn->bits.authneg = FALSE;
 
   Curl_safefree(conn->allocptr.ref);
   if(data->change.referer && !checkheaders(data, "Referer:"))
@@ -1759,7 +1853,7 @@ CURLcode Curl_http(struct connectdata *conn)
     switch(httpreq) {
 
     case HTTPREQ_POST_FORM:
-      if(!http->sendit) {
+      if(!http->sendit || conn->bits.authneg) {
         /* nothing to post! */
         result = add_bufferf(req_buffer, "Content-Length: 0\r\n\r\n");
         if(result)
@@ -1857,11 +1951,16 @@ CURLcode Curl_http(struct connectdata *conn)
 
     case HTTPREQ_PUT: /* Let's PUT the data to the server! */
 
-      if((data->set.infilesize>0) && !conn->bits.upload_chunky) {
+      if(conn->bits.authneg)
+        postsize = 0;
+      else
+        postsize = data->set.infilesize;
+
+      if((postsize != -1) && !conn->bits.upload_chunky) {
         /* only add Content-Length if not uploading chunked */
         result = add_bufferf(req_buffer,
-                             "Content-Length: %" FORMAT_OFF_T "\r\n", /* size */
+                             "Content-Length: %" FORMAT_OFF_T "\r\n",
-                             data->set.infilesize );
+                             postsize );
         if(result)
           return result;
       }
@@ -1884,19 +1983,19 @@ CURLcode Curl_http(struct connectdata *conn)
         return result;
 
       /* set the upload size to the progress meter */
-      Curl_pgrsSetUploadSize(data, data->set.infilesize);
+      Curl_pgrsSetUploadSize(data, postsize);
 
       /* this sends the buffer and frees all the buffer resources */
       result = add_buffer_send(req_buffer, conn,
                                &data->info.request_size);
       if(result)
-        failf(data, "Failed sending POST request");
+        failf(data, "Failed sending PUT request");
       else
         /* prepare for transfer */
         result = Curl_Transfer(conn, FIRSTSOCKET, -1, TRUE,
                                &http->readbytecount,
-                               FIRSTSOCKET,
+                               postsize?FIRSTSOCKET:-1,
-                               &http->writebytecount);
+                               postsize?&http->writebytecount:NULL);
       if(result)
         return result;
       break;
@@ -1904,7 +2003,10 @@ CURLcode Curl_http(struct connectdata *conn)
     case HTTPREQ_POST:
       /* this is the simple POST, using x-www-form-urlencoded style */
 
-      /* store the size of the postfields */
+      if(conn->bits.authneg)
+        postsize = 0;
+      else
+        /* figure out the size of the postfields */
         postsize = (data->set.postfieldsize != -1)?
           data->set.postfieldsize:
           (data->set.postfields?(curl_off_t)strlen(data->set.postfields):0);

lib/http.h

@@ -57,13 +57,20 @@ int Curl_http_should_fail(struct connectdata *conn);
    public curl/curl.h header. */
 #define CURLAUTH_PICKNONE (1<<30) /* don't use auth */
 
-/* MAX_INITIAL_POST_SIZE indicates the number of kilobytes that will be sent
+/* MAX_INITIAL_POST_SIZE indicates the number of bytes that will make the POST
-   in the initial part of a multi-part POST message. This is primarily for
+   data get included in the initial data chunk sent to the server. If the
-   OpenVMS where the maximum number of bytes allowed per I/O is 64K.  For
+   data is larger than this, it will automatically get split up in multiple
-   other systems that do not define this, the default is (as it was
+   system calls.
-   previously) 100K. */
+
+   This value used to be fairly big (100K), but we must take into account that
+   if the server rejects the POST due for authentication reasons, this data
+   will always be uncondtionally sent and thus it may not be larger than can
+   always be afforded to send twice.
+
+   It must not be greater than 64K to work on VMS.
+*/
 #ifndef MAX_INITIAL_POST_SIZE
-#define MAX_INITIAL_POST_SIZE (100*1024)
+#define MAX_INITIAL_POST_SIZE 1024
 #endif
 
 #endif

lib/transfer.c

@@ -193,6 +193,55 @@ checkhttpprefix(struct SessionHandle *data,
   return FALSE;
 }
 
+/*
+ * Curl_readrewind() rewinds the read stream. This typically (so far) only
+ * used for HTTP POST/PUT with multi-pass authentication when a sending was
+ * denied and a resend is necessary.
+ */
+CURLcode Curl_readrewind(struct connectdata *conn)
+{
+  struct SessionHandle *data = conn->data;
+
+  conn->bits.rewindaftersend = FALSE; /* we rewind now */
+
+  /* We have sent away data. If not using CURLOPT_POSTFIELDS or
+     CURLOPT_HTTPPOST, call app to rewind
+  */
+  if(data->set.postfields ||
+     (data->set.httpreq == HTTPREQ_POST_FORM))
+    ; /* do nothing */
+  else {
+    if(data->set.ioctl) {
+      curlioerr err;
+
+      err = data->set.ioctl(data, CURLIOCMD_RESTARTREAD,
+                            data->set.ioctl_client);
+      infof(data, "the ioctl callback returned %d\n", (int)err);
+
+      if(err) {
+        /* FIXME: convert to a human readable error message */
+        failf(data, "ioctl callback returned error %d\n", (int)err);
+        return CURLE_SEND_FAIL_REWIND;
+      }
+    }
+    else {
+      /* If no CURLOPT_READFUNCTION is used, we know that we operate on a
+         given FILE * stream and we can actually attempt to rewind that
+         ourself with fseek() */
+      if(data->set.fread == (curl_read_callback)fread) {
+        if(-1 != fseek(data->set.in, 0, SEEK_SET))
+          /* successful rewind */
+          return CURLE_OK;
+      }
+
+      /* no callback set or failure aboe, makes us fail at once */
+      failf(data, "necessary data rewind wasn't possible\n");
+      return CURLE_SEND_FAIL_REWIND;
+    }
+  }
+  return CURLE_OK;
+}
+
 
 /*
  * Curl_readwrite() is the low-level function to be called when data is to
@@ -1163,6 +1212,12 @@ CURLcode Curl_readwrite(struct connectdata *conn,
             /* done */
             k->keepon &= ~KEEP_WRITE; /* we're done writing */
             writedone = TRUE;
+
+            if(conn->bits.rewindaftersend) {
+              result = Curl_readrewind(conn);
+              if(result)
+                return result;
+            }
             break;
           }
 

lib/transfer.h

@@ -34,7 +34,7 @@ void Curl_single_fdset(struct connectdata *conn,
                        fd_set *exc_fd_set,
                        int *max_fd);
 CURLcode Curl_readwrite_init(struct connectdata *conn);
-
+CURLcode Curl_readrewind(struct connectdata *conn);
 CURLcode Curl_fillreadbuffer(struct connectdata *conn, int bytes, int *nreadp);
 
 /* This sets up a forthcoming transfer */

lib/url.c

@@ -1099,6 +1099,18 @@ CURLcode Curl_setopt(struct SessionHandle *data, CURLoption option, ...)
       /* When set to NULL, reset to our internal default function */
       data->set.fread = (curl_read_callback)fread;
     break;
+  case CURLOPT_IOCTLFUNCTION:
+    /*
+     * I/O control callback. Might be NULL.
+     */
+    data->set.ioctl = va_arg(param, curl_ioctl_callback);
+    break;
+  case CURLOPT_IOCTLDATA:
+    /*
+     * I/O control data pointer. Might be NULL.
+     */
+    data->set.ioctl_client = va_arg(param, void *);
+    break;
   case CURLOPT_SSLCERT:
     /*
      * String that holds file name of the SSL certificate to use

lib/urldata.h

@@ -319,9 +319,14 @@ struct ConnectBits {
                          This is implicit when SSL-protocols are used through
                          proxies, but can also be enabled explicitly by
                          apps */
-  bool authprobe;     /* set TRUE when this transfer is done to probe for auth
+  bool authneg;       /* TRUE when the auth phase has started, which means
-                         types, as when asking for "any" type when speaking
+                         that we are creating a request with an auth header,
-                         HTTP */
+                         but it is not the final request in the auth
+                         negotiation. */
+  bool rewindaftersend;/* TRUE when the sending couldn't be stopped even
+                          though it will be discarded. When the whole send
+                          operation is done, we must call the data rewind
+                          callback. */
 };
 
 struct hostname {
@@ -696,6 +701,9 @@ struct auth {
                  resource */
   bool done;  /* TRUE when the auth phase is done and ready to do the *actual*
                  request */
+  bool multi; /* TRUE if this is not yet authenticated but within the auth
+                 multipass negotiation */
+
 };
 
 struct UrlState {
@@ -827,7 +835,9 @@ struct UserDefined {
   curl_read_callback fread;          /* function that reads the input */
   curl_progress_callback fprogress;  /* function for progress information */
   curl_debug_callback fdebug;      /* function that write informational data */
+  curl_ioctl_callback ioctl;       /* function for I/O control */
   void *progress_client; /* pointer to pass to the progress callback */
+  void *ioctl_client;   /* pointer to pass to the ioctl callback */
   long timeout;         /* in seconds, 0 means no timeout */
   long connecttimeout;  /* in seconds, 0 means no timeout */
   long ftp_response_timeout; /* in seconds, 0 means no timeout */

packages/vms/config-vms.h_with_ssl

@@ -15,12 +15,6 @@
 #endif
 #endif
 
-/* Define to set number of kilobytes in initial POST message. On VMS systems
-   this is important as the default is 100 and the maximum amount of data
-   transferable in a VMS $QIO is 64K. All VMS versions prior to Alpha/I64
-   V8.2 and TCP/IP V5.5 are affected by this. */
-#define MAX_INITIAL_POST_SIZE (60*1024)
-
 /* Define if you have the ANSI C header files.  */
 #define STDC_HEADERS 1
 

packages/vms/config-vms.h_without_ssl

@@ -15,12 +15,6 @@
 #endif
 #endif
 
-/* Define to set the number of kilobytes per POST message. On VMS systems
-   this is important as the default is 100 and the maximum amount of data
-   transferable in a VMS QIO is 64K. All VMS versions prior to Alpha/I64 V8.2
-   and TCP/IP V5.5 are affected by this. */
-#define MAX_INITIAL_POST_SIZE (60*1024)
-
 /* Define if you have the ANSI C header files.  */
 #define STDC_HEADERS 1
 

src/main.c

@@ -2420,8 +2420,30 @@ struct InStruct {
   struct Configurable *config;
 };
 
+static curlioerr my_ioctl(CURL *handle, curliocmd cmd, void *userp)
+{
+  struct InStruct *in=(struct InStruct *)userp;
+  (void)handle; /* not used in here */
+
+  switch(cmd) {
+  case CURLIOCMD_RESTARTREAD:
+    /* mr libcurl kindly asks as to rewind the read data stream to start */
+    if(-1 == fseek(in->stream, 0, SEEK_SET))
+      /* couldn't rewind, the reason is in errno but errno is just not
+         portable enough and we don't actually care that much why we failed. */
+      return CURLIOE_FAILRESTART;
+
+    break;
+
+  default: /* ignore unknown commands */
+    return CURLIOE_UNKNOWNCMD;
+  }
+  return CURLIOE_OK;
+}
+
 static int my_fread(void *buffer, size_t sz, size_t nmemb, void *userp)
 {
+  int rc;
   struct InStruct *in=(struct InStruct *)userp;
   struct Configurable *config = in->config;
   curl_off_t size = (curl_off_t)(sz * nmemb);  /* typecast to prevent warnings
@@ -2480,7 +2502,11 @@ static int my_fread(void *buffer, size_t sz, size_t nmemb, void *userp)
     config->lastsendsize = sz*nmemb;
   }
 
-  return fread(buffer, sz, nmemb, in->stream);
+  rc = fread(buffer, sz, nmemb, in->stream);
+#if 0
+  fprintf(stderr, "CALLBACK returning %d bytes data\n", (int)rc);
+#endif
+  return rc;
 }
 
 struct ProgressData {
@@ -3333,6 +3359,10 @@ operate(struct Configurable *config, int argc, char *argv[])
         /* what call to read */
         curl_easy_setopt(curl, CURLOPT_READFUNCTION, my_fread);
 
+        /* libcurl 7.12.3 business: */
+        curl_easy_setopt(curl, CURLOPT_IOCTLDATA, &input);
+        curl_easy_setopt(curl, CURLOPT_IOCTLFUNCTION, my_ioctl);
+
         if(config->recvpersecond) {
           /* tell libcurl to use a smaller sized buffer as it allows us to
              make better sleeps! 7.9.9 stuff! */

tests/data/test154

@@ -72,10 +72,12 @@ four is the number of lines
 ^User-Agent:.*
 </strip>
 <protocol>
-HEAD /154 HTTP/1.1
+PUT /154 HTTP/1.1
 Host: 127.0.0.1:%HTTPPORT
 Pragma: no-cache
 Accept: */*
+Content-Length: 85
+Expect: 100-continue
 
 PUT /154 HTTP/1.1
 Authorization: Digest username="testuser", realm="gimme all yer s3cr3ts", nonce="11223344", uri="/154", response="b71551e12d1c456e47d8388ecb2edeca"

tests/data/test155

@@ -90,16 +90,20 @@ four is the number of lines
 ^User-Agent:.*
 </strip>
 <protocol>
-HEAD /155 HTTP/1.1
+PUT /155 HTTP/1.1
 Host: 127.0.0.1:%HTTPPORT
 Pragma: no-cache
 Accept: */*
+Content-Length: 85
+Expect: 100-continue
 
-HEAD /155 HTTP/1.1
+PUT /155 HTTP/1.1
 Authorization: NTLM TlRMTVNTUAABAAAAAgIAAAAAAAAgAAAAAAAAACAAAAA=
 Host: 127.0.0.1:%HTTPPORT
 Pragma: no-cache
 Accept: */*
+Content-Length: 0
+Expect: 100-continue
 
 PUT /155 HTTP/1.1
 Authorization: NTLM TlRMTVNTUAADAAAAGAAYAEgAAAAYABgAYAAAAAAAAABAAAAACAAIAEAAAAAAAAAASAAAAAAAAAB4AAAAAYIAAHRlc3R1c2VyWmRDApEJkUyGOPS3DjvASModEeW/N/FBqYVyF4y6/y/7F6qmEQ7lXjXFF3tH1145

tests/data/test156

@@ -58,11 +58,6 @@ four is the number of lines
 ^User-Agent:.*
 </strip>
 <protocol>
-HEAD /156 HTTP/1.1
-Host: 127.0.0.1:%HTTPPORT
-Pragma: no-cache
-Accept: */*
-
 PUT /156 HTTP/1.1
 User-Agent: curl/7.10.5 (i686-pc-linux-gnu) libcurl/7.10.5 OpenSSL/0.9.7a ipv6 zlib/1.1.3
 Host: 127.0.0.1:%HTTPPORT

tests/data/test170

@@ -24,12 +24,13 @@ http://a.galaxy.far.far.away/170 --proxy http://%HOSTIP:%HTTPPORT --proxy-user f
 ^User-Agent: curl/.*
 </strip>
 <protocol>
-HEAD http://a.galaxy.far.far.away/170 HTTP/1.1
+POST http://a.galaxy.far.far.away/170 HTTP/1.1
 Proxy-Authorization: NTLM TlRMTVNTUAABAAAAAgIAAAAAAAAgAAAAAAAAACAAAAA=
 User-Agent: curl/7.12.0-CVS (i686-pc-linux-gnu) libcurl/7.12.0-CVS OpenSSL/0.9.6b ipv6 zlib/1.1.4 libidn/0.4.3
 Host: a.galaxy.far.far.away
 Pragma: no-cache
 Accept: */*
+Content-Length: 0
 
 </protocol>
 # 52 is CURLE_GOT_NOTHING

tests/data/test174

@@ -1,35 +1,13 @@
 # Server-side
 <reply>
 <data>
-HTTP/1.1 200 beng swsclose swsbounce
+HTTP/1.1 200 beng swsclose
 Server: Microsoft-IIS/6.0
 Authentication-Info: Passport1.4 tname=MSPAuth,tname=MSPProf,tname=MSPConsent,tname=MSPSecAuth
 Content-Type: text/html; charset=iso-8859-1
 
 This is not the real page
 </data>
-
-<data1>
-HTTP/1.1 200 moo swsclose
-Server: Microsoft-IIS/6.0
-Content-Type: text/html; charset=iso-8859-1
-
-content for you
-</data1>
-
-<datacheck>
-HTTP/1.1 200 beng swsclose swsbounce
-Server: Microsoft-IIS/6.0
-Authentication-Info: Passport1.4 tname=MSPAuth,tname=MSPProf,tname=MSPConsent,tname=MSPSecAuth
-Content-Type: text/html; charset=iso-8859-1
-
-HTTP/1.1 200 moo swsclose
-Server: Microsoft-IIS/6.0
-Content-Type: text/html; charset=iso-8859-1
-
-content for you
-</datacheck>
-
 </reply>
 
 # Client-side
@@ -52,12 +30,6 @@ http://%HOSTIP:%HTTPPORT/174 -u testuser:testpass --anyauth -d "junkelijunk"
 ^User-Agent:.*
 </strip>
 <protocol nonewline=yes>
-HEAD /174 HTTP/1.1
-User-Agent: curl/7.12.1-CVS (i686-pc-linux-gnu) libcurl/7.12.1-CVS OpenSSL/0.9.6b ipv6 zlib/1.1.4 GSS libidn/0.4.6
-Host: 127.0.0.1:%HTTPPORT
-Pragma: no-cache
-Accept: */*
-
 POST /174 HTTP/1.1
 User-Agent: curl/7.12.1-CVS (i686-pc-linux-gnu) libcurl/7.12.1-CVS OpenSSL/0.9.6b ipv6 zlib/1.1.4 GSS libidn/0.4.6
 Host: 127.0.0.1:%HTTPPORT

tests/data/test175

@@ -52,11 +52,13 @@ http://%HOSTIP:%HTTPPORT/175 -u auser:apasswd --digest -d "junkelijunk"
 ^User-Agent:.*
 </strip>
 <protocol nonewline=yes>
-HEAD /175 HTTP/1.1
+POST /175 HTTP/1.1
 User-Agent: curl/7.12.1-CVS (i686-pc-linux-gnu) libcurl/7.12.1-CVS OpenSSL/0.9.6b ipv6 zlib/1.1.4 GSS libidn/0.4.6
 Host: 127.0.0.1:%HTTPPORT
 Pragma: no-cache
 Accept: */*
+Content-Length: 0
+Content-Type: application/x-www-form-urlencoded
 
 POST /175 HTTP/1.1
 User-Agent: curl/7.12.1-CVS (i686-pc-linux-gnu) libcurl/7.12.1-CVS OpenSSL/0.9.6b ipv6 zlib/1.1.4 GSS libidn/0.4.6

tests/data/test176

@@ -56,12 +56,14 @@ http://%HOSTIP:%HTTPPORT/176 -u auser:apasswd --ntlm -d "junkelijunk"
 ^User-Agent:.*
 </strip>
 <protocol nonewline=yes>
-HEAD /176 HTTP/1.1
+POST /176 HTTP/1.1
 Authorization: NTLM TlRMTVNTUAABAAAAAgIAAAAAAAAgAAAAAAAAACAAAAA=
 User-Agent: curl/7.12.1-CVS (i686-pc-linux-gnu) libcurl/7.12.1-CVS OpenSSL/0.9.6b ipv6 zlib/1.1.4 GSS libidn/0.4.6
 Host: 127.0.0.1:%HTTPPORT
 Pragma: no-cache
 Accept: */*
+Content-Length: 0
+Content-Type: application/x-www-form-urlencoded
 
 POST /176 HTTP/1.1
 User-Agent: curl/7.12.1-CVS (i686-pc-linux-gnu) libcurl/7.12.1-CVS OpenSSL/0.9.6b ipv6 zlib/1.1.4 GSS libidn/0.4.6

tests/data/test177

@@ -29,11 +29,13 @@ http://%HOSTIP:%HTTPPORT/177 -u auser:apasswd --digest -d "junkelijunk"
 ^User-Agent:.*
 </strip>
 <protocol>
-HEAD /177 HTTP/1.1
+POST /177 HTTP/1.1
 User-Agent: curl/7.12.1-CVS (i686-pc-linux-gnu) libcurl/7.12.1-CVS OpenSSL/0.9.6b ipv6 zlib/1.1.4 GSS libidn/0.4.6
 Host: 127.0.0.1:%HTTPPORT
 Pragma: no-cache
 Accept: */*
+Content-Length: 0
+Content-Type: application/x-www-form-urlencoded
 
 </protocol>
 </verify>

tests/data/test88

@@ -66,10 +66,12 @@ four is the number of lines
 ^User-Agent:.*
 </strip>
 <protocol>
-HEAD /88 HTTP/1.1
+PUT /88 HTTP/1.1
 Host: 127.0.0.1:%HTTPPORT
 Pragma: no-cache
 Accept: */*
+Content-Length: 0
+Expect: 100-continue
 
 PUT /88 HTTP/1.1
 Authorization: Digest username="testuser", realm="testrealm", nonce="1053604145", uri="/88", response="78a49fa53d0c228778297687d4168e71"