David McCreedy brought line end conversions when doing FTP ASCII

transfers. They are done on non-windows systems and translate CRLF to LF.

CHANGES

@@ -6,6 +6,17 @@
 
                                   Changelog
 
+Daniel (26 April 2006)
+- David McCreedy brought initial line end conversions when doing FTP ASCII
+  transfers. They are done on non-windows systems and translate CRLF to LF.
+
+  I modified the 15 LIST-using test cases accordingly. The downside is that now
+  we'll have even more trouble to get the tests to run on Windows since they
+  should get CRLF newlines left intact which the *nix versions don't. I figure
+  the only sane thing to do is to add some kind of [newline] macro for the test
+  case files and have them expanded to the proper native line ending when the
+  test cases are run. This is however left to implement.
+
 Daniel (25 April 2006)
 - Paul Querna fixed libcurl to better deal with deflate content encoding
   when the stream (wrongly) lacks a proper zlib header. This seems to be the

RELEASE-NOTES

@@ -11,6 +11,7 @@ Curl and libcurl 7.15.4
 
 This release includes the following changes:
 
+ o line end conversions for FTP ASCII transfers
  o curl_multi_socket() API added (still mostly untested)
  o conversion callback options for EBCDIC <=> ASCII conversions
  o added CURLINFO_FTP_ENTRY_PATH
@@ -37,6 +38,7 @@ Other curl-related news since the previous public release:
 
  o tclcurl 0.15.3 was released:
    http://personal1.iddeo.es/andresgarci/tclcurl/english/
+ o http://curl.usphp.com/ is a new mirror in Florida, US
 
 This release would not have looked like this without help, code, reports and
 advice from friends like these:

TODO-RELEASE

@@ -1,6 +1,5 @@
 To get fixed in 7.15.4 (planned release: June 2006)
 ======================
 
-65 - curl_multi_socket() added but not extensively tested nor particularly
+66 - 
-     documented or pushed for.
 

docs/KNOWN_BUGS

@@ -66,6 +66,8 @@ may have been fixed since this was written!
      specification).  The receiver will convert the data from the standard
      form to his own internal form.
 
+   Since 7.15.4 at least line endings are converted.
+
 19. FTP 3rd party transfers with the multi interface doesn't work. Test:
   define CURL_MULTIEASY, rebuild curl, run test case 230 - 232.
 

lib/ftp.c

@@ -2994,6 +2994,13 @@ CURLcode Curl_ftp_done(struct connectdata *conn, CURLcode status)
   }
   else {
     if((-1 != conn->size) && (conn->size != *ftp->bytecountp) &&
+#ifdef CURL_DO_LINEEND_CONV
+       /* Most FTP servers don't adjust their file SIZE response for CRLFs, so
+        * we'll check to see if the discrepancy can be explained by the number
+        * of CRLFs we've changed to LFs.
+        */
+       ((conn->size + data->state.crlf_conversions) != *ftp->bytecountp) &&
+#endif /* CURL_DO_LINEEND_CONV */
        (conn->maxdownload != *ftp->bytecountp)) {
       failf(data, "Received only partial file: %" FORMAT_OFF_T " bytes",
             *ftp->bytecountp);

lib/sendf.c

@@ -5,7 +5,7 @@
  *                            | (__| |_| |  _ <| |___
  *                             \___|\___/|_| \_\_____|
  *
- * Copyright (C) 1998 - 2005, Daniel Stenberg, <daniel@haxx.se>, et al.
+ * Copyright (C) 1998 - 2006, Daniel Stenberg, <daniel@haxx.se>, et al.
  *
  * This software is licensed as described in the file COPYING, which
  * you should have received as part of this distribution. The terms
@@ -139,6 +139,89 @@ void curl_slist_free_all(struct curl_slist *list)
   } while (next);
 }
 
+#ifdef CURL_DO_LINEEND_CONV
+/*
+ * convert_lineends() changes CRLF (\r\n) end-of-line markers to a single LF
+ * (\n), with special processing for CRLF sequences that are split between two
+ * blocks of data.  Remaining, bare CRs are changed to LFs.  The possibly new
+ * size of the data is returned.
+ */
+static size_t convert_lineends(struct SessionHandle *data,
+                               char *startPtr, size_t size)
+{
+  char *inPtr, *outPtr;
+
+  /* sanity check */
+  if ((startPtr == NULL) || (size < 1)) {
+    return(size);
+  }
+
+  if (data->state.prev_block_had_trailing_cr == TRUE) {
+    /* The previous block of incoming data
+       had a trailing CR, which was turned into a LF. */
+    if (*startPtr == '\n') {
+      /* This block of incoming data starts with the
+         previous block's LF so get rid of it */
+      memcpy(startPtr, startPtr+1, size-1);
+      size--;
+      /* and it wasn't a bare CR but a CRLF conversion instead */
+      data->state.crlf_conversions++;
+    }
+    data->state.prev_block_had_trailing_cr = FALSE; /* reset the flag */
+  }
+
+  /* find 1st CR, if any */
+  inPtr = outPtr = memchr(startPtr, '\r', size);
+  if (inPtr) {
+    /* at least one CR, now look for CRLF */
+    while (inPtr < (startPtr+size-1)) {
+      /* note that it's size-1, so we'll never look past the last byte */
+      if (memcmp(inPtr, "\r\n", 2) == 0) {
+        /* CRLF found, bump past the CR and copy the NL */
+        inPtr++;
+        *outPtr = *inPtr;
+        /* keep track of how many CRLFs we converted */
+        data->state.crlf_conversions++;
+      }
+      else {
+        if (*inPtr == '\r') {
+          /* lone CR, move LF instead */
+          *outPtr = '\n';
+        }
+        else {
+          /* not a CRLF nor a CR, just copy whatever it is */
+          *outPtr = *inPtr;
+        }
+      }
+      outPtr++;
+      inPtr++;
+    } /* end of while loop */
+
+    if (inPtr < startPtr+size) {
+      /* handle last byte */
+      if (*inPtr == '\r') {
+        /* deal with a CR at the end of the buffer */
+        *outPtr = '\n'; /* copy a NL instead */
+        /* note that a CRLF might be split across two blocks */
+        data->state.prev_block_had_trailing_cr = TRUE;
+      }
+      else {
+        /* copy last byte */
+        *outPtr = *inPtr;
+      }
+      outPtr++;
+      inPtr++;
+    }
+    if (outPtr < startPtr+size) {
+      /* tidy up by null terminating the now shorter data */
+      *outPtr = '\0';
+    }
+    return(outPtr - startPtr);
+  }
+  return(size);
+}
+#endif /* CURL_DO_LINEEND_CONV */
+
 /* Curl_infof() is for info message along the way */
 
 void Curl_infof(struct SessionHandle *data, const char *fmt, ...)
@@ -294,36 +377,36 @@ CURLcode Curl_client_write(struct SessionHandle *data,
   if(0 == len)
     len = strlen(ptr);
 
-#ifdef CURL_DOES_CONVERSIONS
   if(type & CLIENTWRITE_BODY) {
     if(data->ftp_in_ascii_mode) {
+#ifdef CURL_DOES_CONVERSIONS
       /* convert from the network encoding */
       size_t rc;
       rc = Curl_convert_from_network(data, ptr, len);
       /* Curl_convert_from_network calls failf if unsuccessful */
-      if(rc != CURLE_OK) {
+      if(rc != CURLE_OK)
-        return(rc);
+        return rc;
-      }
+#endif /* CURL_DOES_CONVERSIONS */
+
+#ifdef CURL_DO_LINEEND_CONV
+      /* convert end-of-line markers */
+      len = convert_lineends(data, ptr, len);
+#endif /* CURL_DO_LINEEND_CONV */
     }
+    /* If the previous block of data ended with CR and this block of data is
+       just a NL, then the length might be zero */
     if (len) {
       wrote = data->set.fwrite(ptr, 1, len, data->set.out);
-    } else {
+    }
+    else {
       wrote = len;
     }
+
     if(wrote != len) {
       failf (data, "Failed writing body");
       return CURLE_WRITE_ERROR;
     }
   }
-#else
-  if(type & CLIENTWRITE_BODY) {
-    wrote = data->set.fwrite(ptr, 1, len, data->set.out);
-    if(wrote != len) {
-      failf (data, "Failed writing body");
-      return CURLE_WRITE_ERROR;
-    }
-  }
-#endif /* CURL_DOES_CONVERSIONS */
 
   if((type & CLIENTWRITE_HEADER) &&
      (data->set.fwrite_header || data->set.writeheader) ) {

lib/transfer.c

@@ -1281,17 +1281,33 @@ CURLcode Curl_readwrite(struct connectdata *conn,
           conn->upload_present = nread;
 
           /* convert LF to CRLF if so asked */
+#ifdef CURL_DO_LINEEND_CONV
+          /* always convert if we're FTPing in ASCII mode */
+          if ((data->set.crlf) || (data->ftp_in_ascii_mode)) {
+#else
           if (data->set.crlf) {
+#endif /* CURL_DO_LINEEND_CONV */
               if(data->state.scratch == NULL)
                 data->state.scratch = malloc(2*BUFSIZE);
               if(data->state.scratch == NULL) {
                 failf (data, "Failed to alloc scratch buffer!");
                 return CURLE_OUT_OF_MEMORY;
               }
+              /*
+               * ASCII/EBCDIC Note: This is presumably a text (not binary)
+               * transfer so the data should already be in ASCII.
+               * That means the hex values for ASCII CR (0x0d) & LF (0x0a)
+               * must be used instead of the escape sequences \r & \n.
+               */
             for(i = 0, si = 0; i < nread; i++, si++) {
               if (conn->upload_fromhere[i] == 0x0a) {
                 data->state.scratch[si++] = 0x0d;
                 data->state.scratch[si] = 0x0a;
+                if (!data->set.crlf) {
+                  /* we're here only because FTP is in ASCII mode...
+                     bump infilesize for the LF we just added */
+                  data->set.infilesize++;
+                }
               }
               else
                 data->state.scratch[si] = conn->upload_fromhere[i];
@@ -1417,6 +1433,13 @@ CURLcode Curl_readwrite(struct connectdata *conn,
 
     if(!(conn->bits.no_body) && (conn->size != -1) &&
        (k->bytecount != conn->size) &&
+#ifdef CURL_DO_LINEEND_CONV
+       /* Most FTP servers don't adjust their file SIZE response for CRLFs,
+          so we'll check to see if the discrepancy can be explained
+          by the number of CRLFs we've changed to LFs.
+        */
+       (k->bytecount != (conn->size + data->state.crlf_conversions)) &&
+#endif /* CURL_DO_LINEEND_CONV */
        !conn->newurl) {
       failf(data, "transfer closed with %" FORMAT_OFF_T
             " bytes remaining to read",

lib/urldata.h

@@ -936,6 +936,15 @@ struct UrlState {
 
   /* a place to store the most recenlty set FTP entrypath */
   char *most_recent_ftp_entrypath;
+
+#ifndef WIN32
+/* do FTP line-end conversions on most platforms */
+#define CURL_DO_LINEEND_CONV
+  /* for FTP downloads: track CRLF sequences that span blocks */
+  bool prev_block_had_trailing_cr;
+  /* for FTP downloads: how many CRLFs did we converted to LFs? */
+  size_t crlf_conversions;
+#endif
 };
 
 

tests/data/test100

@@ -8,7 +8,9 @@ LIST
 #
 # Server-side
 <reply>
-<data>
+# When doing LIST, we get the default list output hard-coded in the test
+# FTP server
+<datacheck>
 total 20
 drwxr-xr-x   8 98       98           512 Oct 22 13:06 .
 drwxr-xr-x   8 98       98           512 Oct 22 13:06 ..
@@ -20,7 +22,7 @@ drwxrwxrwx   2 98       98           512 May 29 16:04 download.html
 dr-xr-xr-x   2 0        1            512 Nov 30  1995 etc
 drwxrwxrwx   2 98       1            512 Oct 30 14:33 pub
 dr-xr-xr-x   5 0        1            512 Oct  1  1997 usr
-</data>
+</datacheck>
 </reply>
 
 #

tests/data/test101

@@ -7,7 +7,9 @@ LIST
 </info>
 # Server-side
 <reply>
-<data>
+# When doing LIST, we get the default list output hard-coded in the test
+# FTP server
+<datacheck>
 total 20
 drwxr-xr-x   8 98       98           512 Oct 22 13:06 .
 drwxr-xr-x   8 98       98           512 Oct 22 13:06 ..
@@ -19,7 +21,7 @@ drwxrwxrwx   2 98       98           512 May 29 16:04 download.html
 dr-xr-xr-x   2 0        1            512 Nov 30  1995 etc
 drwxrwxrwx   2 98       1            512 Oct 30 14:33 pub
 dr-xr-xr-x   5 0        1            512 Oct  1  1997 usr
-</data>
+</datacheck>
 </reply>
 
 # Client-side

tests/data/test130

@@ -9,7 +9,9 @@ netrc
 #
 # Server-side
 <reply>
-<data>
+# When doing LIST, we get the default list output hard-coded in the test
+# FTP server
+<datacheck>
 total 20
 drwxr-xr-x   8 98       98           512 Oct 22 13:06 .
 drwxr-xr-x   8 98       98           512 Oct 22 13:06 ..
@@ -21,7 +23,7 @@ drwxrwxrwx   2 98       98           512 May 29 16:04 download.html
 dr-xr-xr-x   2 0        1            512 Nov 30  1995 etc
 drwxrwxrwx   2 98       1            512 Oct 30 14:33 pub
 dr-xr-xr-x   5 0        1            512 Oct  1  1997 usr
-</data>
+</datacheck>
 </reply>
 
 #

tests/data/test131

@@ -8,8 +8,11 @@ netrca
 </info>
 #
 # Server-side
-<reply name="1">
+<reply>
-<data>
+#
+# When doing LIST, we get the default list output hard-coded in the test
+# FTP server
+<datacheck>
 total 20
 drwxr-xr-x   8 98       98           512 Oct 22 13:06 .
 drwxr-xr-x   8 98       98           512 Oct 22 13:06 ..
@@ -21,7 +24,7 @@ drwxrwxrwx   2 98       98           512 May 29 16:04 download.html
 dr-xr-xr-x   2 0        1            512 Nov 30  1995 etc
 drwxrwxrwx   2 98       1            512 Oct 30 14:33 pub
 dr-xr-xr-x   5 0        1            512 Oct  1  1997 usr
-</data>
+</datacheck>
 </reply>
 
 #

tests/data/test132

@@ -9,7 +9,9 @@ netrc
 #
 # Server-side
 <reply>
-<data>
+# When doing LIST, we get the default list output hard-coded in the test
+# FTP server
+<datacheck>
 total 20
 drwxr-xr-x   8 98       98           512 Oct 22 13:06 .
 drwxr-xr-x   8 98       98           512 Oct 22 13:06 ..
@@ -21,7 +23,7 @@ drwxrwxrwx   2 98       98           512 May 29 16:04 download.html
 dr-xr-xr-x   2 0        1            512 Nov 30  1995 etc
 drwxrwxrwx   2 98       1            512 Oct 30 14:33 pub
 dr-xr-xr-x   5 0        1            512 Oct  1  1997 usr
-</data>
+</datacheck>
 </reply>
 
 #

tests/data/test133

@@ -8,8 +8,10 @@ netrc
 </info>
 #
 # Server-side
-<reply name="1">
+<reply>
-<data>
+# When doing LIST, we get the default list output hard-coded in the test
+# FTP server
+<datacheck>
 total 20
 drwxr-xr-x   8 98       98           512 Oct 22 13:06 .
 drwxr-xr-x   8 98       98           512 Oct 22 13:06 ..
@@ -21,8 +23,7 @@ drwxrwxrwx   2 98       98           512 May 29 16:04 download.html
 dr-xr-xr-x   2 0        1            512 Nov 30  1995 etc
 drwxrwxrwx   2 98       1            512 Oct 30 14:33 pub
 dr-xr-xr-x   5 0        1            512 Oct  1  1997 usr
-</data>
+</datacheck>
-</reply>
 
 #
 # Client-side

tests/data/test134

@@ -8,8 +8,10 @@ netrc
 </info>
 #
 # Server-side
-<reply name="1">
+<reply>
-<data>
+# When doing LIST, we get the default list output hard-coded in the test
+# FTP server
+<datacheck>
 total 20
 drwxr-xr-x   8 98       98           512 Oct 22 13:06 .
 drwxr-xr-x   8 98       98           512 Oct 22 13:06 ..
@@ -21,7 +23,7 @@ drwxrwxrwx   2 98       98           512 May 29 16:04 download.html
 dr-xr-xr-x   2 0        1            512 Nov 30  1995 etc
 drwxrwxrwx   2 98       1            512 Oct 30 14:33 pub
 dr-xr-xr-x   5 0        1            512 Oct  1  1997 usr
-</data>
+</datacheck>
 </reply>
 
 #

tests/data/test215

@@ -1,5 +1,7 @@
 # Server-side
 <reply>
+# When doing LIST, we get the default list output hard-coded in the test
+# FTP server
 <datacheck>
 total 20
 drwxr-xr-x   8 98       98           512 Oct 22 13:06 .

tests/data/test250

@@ -1,7 +1,9 @@
 #
 # Server-side
 <reply>
-<data>
+# When doing LIST, we get the default list output hard-coded in the test
+# FTP server
+<datacheck>
 total 20
 drwxr-xr-x   8 98       98           512 Oct 22 13:06 .
 drwxr-xr-x   8 98       98           512 Oct 22 13:06 ..
@@ -13,7 +15,7 @@ drwxrwxrwx   2 98       98           512 May 29 16:04 download.html
 dr-xr-xr-x   2 0        1            512 Nov 30  1995 etc
 drwxrwxrwx   2 98       1            512 Oct 30 14:33 pub
 dr-xr-xr-x   5 0        1            512 Oct  1  1997 usr
-</data>
+</datacheck>
 <servercmd>
 SLOWDOWN
 </servercmd>

tests/data/test251

@@ -1,6 +1,8 @@
 # Server-side
 <reply>
-<data>
+# When doing LIST, we get the default list output hard-coded in the test
+# FTP server
+<datacheck>
 total 20
 drwxr-xr-x   8 98       98           512 Oct 22 13:06 .
 drwxr-xr-x   8 98       98           512 Oct 22 13:06 ..
@@ -12,7 +14,7 @@ drwxrwxrwx   2 98       98           512 May 29 16:04 download.html
 dr-xr-xr-x   2 0        1            512 Nov 30  1995 etc
 drwxrwxrwx   2 98       1            512 Oct 30 14:33 pub
 dr-xr-xr-x   5 0        1            512 Oct  1  1997 usr
-</data>
+</datacheck>
 <servercmd>
 SLOWDOWN
 </servercmd>

tests/data/test252

@@ -7,7 +7,9 @@ EPSV
 #
 # Server-side
 <reply>
-<data>
+# When doing LIST, we get the default list output hard-coded in the test
+# FTP server
+<datacheck>
 total 20
 drwxr-xr-x   8 98       98           512 Oct 22 13:06 .
 drwxr-xr-x   8 98       98           512 Oct 22 13:06 ..
@@ -19,7 +21,7 @@ drwxrwxrwx   2 98       98           512 May 29 16:04 download.html
 dr-xr-xr-x   2 0        1            512 Nov 30  1995 etc
 drwxrwxrwx   2 98       1            512 Oct 30 14:33 pub
 dr-xr-xr-x   5 0        1            512 Oct  1  1997 usr
-</data>
+</datacheck>
 </reply>
 
 #

tests/data/test253

@@ -7,7 +7,9 @@ EPRT
 #
 # Server-side
 <reply>
-<data>
+# When doing LIST, we get the default list output hard-coded in the test
+# FTP server
+<datacheck>
 total 20
 drwxr-xr-x   8 98       98           512 Oct 22 13:06 .
 drwxr-xr-x   8 98       98           512 Oct 22 13:06 ..
@@ -19,7 +21,7 @@ drwxrwxrwx   2 98       98           512 May 29 16:04 download.html
 dr-xr-xr-x   2 0        1            512 Nov 30  1995 etc
 drwxrwxrwx   2 98       1            512 Oct 30 14:33 pub
 dr-xr-xr-x   5 0        1            512 Oct  1  1997 usr
-</data>
+</datacheck>
 </reply>
 
 #

tests/data/test254

@@ -8,7 +8,9 @@ EPSV
 #
 # Server-side
 <reply>
-<data>
+# When doing LIST, we get the default list output hard-coded in the test
+# FTP server
+<datacheck>
 total 20
 drwxr-xr-x   8 98       98           512 Oct 22 13:06 .
 drwxr-xr-x   8 98       98           512 Oct 22 13:06 ..
@@ -20,7 +22,7 @@ drwxrwxrwx   2 98       98           512 May 29 16:04 download.html
 dr-xr-xr-x   2 0        1            512 Nov 30  1995 etc
 drwxrwxrwx   2 98       1            512 Oct 30 14:33 pub
 dr-xr-xr-x   5 0        1            512 Oct  1  1997 usr
-</data>
+</datacheck>
 </reply>
 
 #

tests/data/test255

@@ -8,7 +8,9 @@ EPRT
 #
 # Server-side
 <reply>
-<data>
+# When doing LIST, we get the default list output hard-coded in the test
+# FTP server
+<datacheck>
 total 20
 drwxr-xr-x   8 98       98           512 Oct 22 13:06 .
 drwxr-xr-x   8 98       98           512 Oct 22 13:06 ..
@@ -20,7 +22,7 @@ drwxrwxrwx   2 98       98           512 May 29 16:04 download.html
 dr-xr-xr-x   2 0        1            512 Nov 30  1995 etc
 drwxrwxrwx   2 98       1            512 Oct 30 14:33 pub
 dr-xr-xr-x   5 0        1            512 Oct  1  1997 usr
-</data>
+</datacheck>
 </reply>
 
 #

tests/data/test521

@@ -8,7 +8,9 @@ CURLOPT_PORT
 #
 # Server-side
 <reply>
-<data>
+# When doing LIST, we get the default list output hard-coded in the test
+# FTP server
+<datacheck>
 total 20
 drwxr-xr-x   8 98       98           512 Oct 22 13:06 .
 drwxr-xr-x   8 98       98           512 Oct 22 13:06 ..
@@ -20,7 +22,7 @@ drwxrwxrwx   2 98       98           512 May 29 16:04 download.html
 dr-xr-xr-x   2 0        1            512 Nov 30  1995 etc
 drwxrwxrwx   2 98       1            512 Oct 30 14:33 pub
 dr-xr-xr-x   5 0        1            512 Oct  1  1997 usr
-</data>
+</datacheck>
 </reply>
 
 #