curl/lib
Daniel Stenberg 3c6d3b69c2 1 - allow DICT with properly URL-escaped words, like using %20 for spaces
2 - properly escape certain letters within a DICT word to comply to the RFC2229
2006-05-10 11:44:31 +00:00
..
.cvsignore ignore curllib.dsp 2004-08-10 10:43:41 +00:00
Makefile.Watcom Added splay.c. 2006-04-12 14:01:21 +00:00
Makefile.am added README.multi_socket 2006-04-10 14:54:12 +00:00
Makefile.b32 Static lib is libcurl.lib and import lib libcurl_imp.lib. 2004-11-14 13:48:15 +00:00
Makefile.inc First curl_multi_socket() commit. Should primarily be considered as an internal 2006-04-10 15:00:53 +00:00
Makefile.m32 OpenSSL updates; get CA_BUNDLE from env. Assume no 2004-12-19 11:52:31 +00:00
Makefile.netware removed -fpack-struct because gcc4 seems to know its obsolete and warns... 2006-04-20 10:26:51 +00:00
Makefile.riscos GnuTLS updates 2005-04-07 21:10:31 +00:00
Makefile.vc6 added splay 2006-04-12 18:12:46 +00:00
README.ares up-to-date with reality 2004-04-06 07:59:11 +00:00
README.curlx updated with more and new info 2004-04-30 08:51:19 +00:00
README.encoding Fixed a lingering omission of gzip support. 2005-12-08 18:59:19 +00:00
README.hostip Added README.hostip 2005-01-14 13:43:29 +00:00
README.httpauth HTTP "auth done right". See lib/README.httpauth 2004-11-24 16:11:35 +00:00
README.memoryleak CURLDEBUG, not MALLOCDEBUG 2003-07-22 10:00:37 +00:00
README.multi_socket each socket is used by exactly one easy handle, but of course each easy handle 2006-04-21 13:40:07 +00:00
amigaos.c Made the copyright year match the latest modification's year. 2006-01-09 13:17:14 +00:00
amigaos.h Made the copyright year match the latest modification's year. 2006-01-09 13:17:14 +00:00
arpa_telnet.h Make some more arrays of pointers const. 2004-12-20 18:23:43 +00:00
base64.c Updated the copyright year since changes have been this year. 2005-03-31 07:02:02 +00:00
base64.h Updated the copyright year since changes have been this year. 2005-03-31 07:02:02 +00:00
ca-bundle.crt Removed the "TC TrustCenter, Germany, Class 0 CA." certificate: 2003-03-24 11:06:57 +00:00
config-amigaos.h Diego Casorran patches to make (lib)curl build fine on Amiga again 2005-07-13 18:06:40 +00:00
config-mac.h Fixed LDAP library file name bug (KNOWN_BUGS #1). configure now auto-detects 2005-03-11 05:28:07 +00:00
config-riscos.h Fixed LDAP library file name bug (KNOWN_BUGS #1). configure now auto-detects 2005-03-11 05:28:07 +00:00
config-tpf.h First commit of David McCreedy's EBCDIC and TPF changes. 2006-04-07 21:50:47 +00:00
config-win32.h Added support for Salford-C under Win32 (scc). HAVE_MALLOC_H and 2006-04-26 17:11:05 +00:00
config-win32ce.h Added support for Salford-C under Win32 (scc). HAVE_MALLOC_H and 2006-04-26 17:11:05 +00:00
config.dj djgpp has <process.h> too. 2006-04-26 17:15:57 +00:00
connect.c Roland Blom filed bug report #1481217 2006-05-04 22:39:47 +00:00
connect.h Roland Blom filed bug report #1481217 2006-05-04 22:39:47 +00:00
content_encoding.c Paul Querna fixed libcurl to better deal with deflate content encoding when 2006-04-25 20:49:40 +00:00
content_encoding.h strip trailing whitespace 2004-08-16 13:24:01 +00:00
cookie.c remove the typecast to long from time_t, since we now store it as curl_off_t 2005-08-17 09:11:27 +00:00
cookie.h Made the copyright year match the latest modification's year. 2006-01-09 13:17:14 +00:00
curllib.dsw Hanno L. Kranzhoff fixed them 2002-06-14 09:36:09 +00:00
curlx.h delete trailing whitespace 2004-05-24 11:57:34 +00:00
dict.c 1 - allow DICT with properly URL-escaped words, like using %20 for spaces 2006-05-10 11:44:31 +00:00
dict.h FTP code turned into state machine. Not completely yet, but a good start. 2005-02-09 13:06:40 +00:00
easy.c First commit of David McCreedy's EBCDIC and TPF changes. 2006-04-07 21:50:47 +00:00
easyif.h First commit of David McCreedy's EBCDIC and TPF changes. 2006-04-07 21:50:47 +00:00
escape.c First commit of David McCreedy's EBCDIC and TPF changes. 2006-04-07 21:50:47 +00:00
escape.h First commit of David McCreedy's EBCDIC and TPF changes. 2006-04-07 21:50:47 +00:00
file.c Roland Blom filed bug report #1481217 2006-05-04 22:39:47 +00:00
file.h FTP code turned into state machine. Not completely yet, but a good start. 2005-02-09 13:06:40 +00:00
formdata.c Fixed signed/unsigned convertion errors in Salford-C. 2006-04-26 17:26:22 +00:00
formdata.h Allow formposting of files larger than what fits in memory by not reading the 2004-06-14 08:51:43 +00:00
ftp.c Roland Blom filed bug report #1481217 2006-05-04 22:39:47 +00:00
ftp.h First curl_multi_socket() commit. Should primarily be considered as an internal 2006-04-10 15:00:53 +00:00
getenv.c Made the copyright year match the latest modification's year. 2006-01-09 13:17:14 +00:00
getinfo.c David McCreedy added CURLINFO_FTP_ENTRY_PATH to export the FTP entry path 2006-03-21 22:30:03 +00:00
getinfo.h updated year in the copyright string 2004-01-07 09:19:33 +00:00
gtls.c additional renames of Curl_ourerrno => Curl_sockerrno 2006-05-05 10:24:27 +00:00
gtls.h GnuTLS support added. There's now a "generic" SSL layer that we use all over 2005-04-07 15:27:13 +00:00
hash.c First curl_multi_socket() commit. Should primarily be considered as an internal 2006-04-10 15:00:53 +00:00
hash.h Use plain structs and not typedef'ed ones in the hash and linked-list code. 2005-01-25 00:06:29 +00:00
hostares.c get the Curl_sockerrno proto 2006-05-05 22:07:01 +00:00
hostasyn.c Roland Blom filed bug report #1481217 2006-05-04 22:39:47 +00:00
hostip.c Roland Blom filed bug report #1481217 2006-05-04 22:39:47 +00:00
hostip.h added docs and removed proto 2006-04-11 07:22:55 +00:00
hostip4.c Use the HAVE_MALLOC_H and HAVE_PROCESS_H defines 2006-04-26 17:23:28 +00:00
hostip6.c additional renames of Curl_ourerrno => Curl_sockerrno 2006-05-05 10:24:27 +00:00
hostsyn.c Roland Blom filed bug report #1481217 2006-05-04 22:39:47 +00:00
hostthre.c Use the HAVE_MALLOC_H and HAVE_PROCESS_H defines 2006-04-26 17:23:28 +00:00
http.c Curl_https_getsock() was OpenSSL-specific and really should not be present 2006-05-05 22:14:40 +00:00
http.h First curl_multi_socket() commit. Should primarily be considered as an internal 2006-04-10 15:00:53 +00:00
http_chunks.c Adrian Schuur added trailer support in the chunked encoding stream. The 2005-07-12 18:15:34 +00:00
http_chunks.h Adrian Schuur added trailer support in the chunked encoding stream. The 2005-07-12 18:15:34 +00:00
http_digest.c Dan Fandrich added the --disable-crypto-auth option to configure to allow 2004-11-12 09:18:14 +00:00
http_digest.h return CURLDIGEST_NOMEM when a memory function fails to deliver 2004-05-13 14:14:03 +00:00
http_negotiate.c Roland Blom filed bug report #1481217 2006-05-04 22:39:47 +00:00
http_negotiate.h updated year in the copyright string 2004-01-07 09:19:33 +00:00
http_ntlm.c readint_le() not needed in USE_WINDOWS_SSPI code. 2006-04-08 11:04:53 +00:00
http_ntlm.h only enable NTLM if HTTP and NTLM is not disabled, and if NTLM is disabled 2005-11-23 11:51:31 +00:00
if2ip.c Diego Casorran patches to make (lib)curl build fine on Amiga again 2005-07-13 18:06:40 +00:00
if2ip.h build fix for Interix 2006-03-04 22:39:31 +00:00
inet_ntoa_r.h provide the proper copyright texts for these 2005-05-26 20:56:25 +00:00
inet_ntop.c Replaced nonstandard u_char and u_int types 2005-12-08 22:59:58 +00:00
inet_ntop.h Yang Tse: fixes the use of Curl_inet_ntop and Curl_inet_pton with no 2005-11-25 22:20:02 +00:00
inet_pton.c Replaced nonstandard u_char and u_int types 2005-12-08 22:59:58 +00:00
inet_pton.h Yang Tse's changes to provide an inet_pton() proto for the platforms who 2005-11-28 20:21:35 +00:00
krb4.c updated source header 2006-01-26 10:39:25 +00:00
krb4.h GnuTLS support added. There's now a "generic" SSL layer that we use all over 2005-04-07 15:27:13 +00:00
ldap.c Use the HAVE_MALLOC_H and HAVE_PROCESS_H defines 2006-04-26 17:23:28 +00:00
ldap.h FTP code turned into state machine. Not completely yet, but a good start. 2005-02-09 13:06:40 +00:00
libcurl.def Yang Tse: msvc7+ has deprecated the 'DESCRIPTION' section in 2005-11-14 07:48:05 +00:00
libcurl.framework.make added new files 2005-04-07 20:56:04 +00:00
libcurl.imp there's an curl_easy_unescape too now 2006-04-18 09:23:03 +00:00
libcurl.plist Matt Veenstra updated to 7.12.3. Starting now, we'll update the version number 2004-12-08 23:09:23 +00:00
libcurl.rc Update copyright. 2005-07-05 18:07:55 +00:00
llist.c Use plain structs and not typedef'ed ones in the hash and linked-list code. 2005-01-25 00:06:29 +00:00
llist.h Use plain structs and not typedef'ed ones in the hash and linked-list code. 2005-01-25 00:06:29 +00:00
makefile.amiga Diego Casorran patches to make (lib)curl build fine on Amiga again 2005-07-13 18:06:40 +00:00
makefile.dj Updated generated dependencies. 2005-05-14 06:04:21 +00:00
md5.c corrected copyright year 2005-05-02 14:33:07 +00:00
md5.h Add 'const' to immutable arrays. 2004-12-15 01:38:25 +00:00
memdebug.c use %p to printf pointers since %x doesn't work properly on tru64 for this 2005-06-30 13:30:23 +00:00
memdebug.h Yang Tse fixed compiler warnings 2005-11-13 23:53:14 +00:00
memory.h removed tabs and trailing whitespace from source 2004-10-06 07:50:18 +00:00
mprintf.c Diego Casorran patches to make (lib)curl build fine on Amiga again 2005-07-13 18:06:40 +00:00
msvcproj.foot new attempt at an improved DSP-file generation 2004-05-11 07:54:55 +00:00
msvcproj.head Use the more correct BUILDING_LIBCURL define instead of CURLLIB_EXPORTS. 2005-08-29 20:56:20 +00:00
multi.c Robson Braga Araujo fixed a memory leak when you added an easy handle to a 2006-05-09 11:33:00 +00:00
multiif.h First curl_multi_socket() commit. Should primarily be considered as an internal 2006-04-10 15:00:53 +00:00
netrc.c corrected copyright year 2005-05-02 14:33:07 +00:00
netrc.h updated year in the copyright string 2004-01-07 09:19:33 +00:00
nwlib.c converted to UNIX format. 2004-06-10 21:20:15 +00:00
parsedate.c fix questionable compare 2005-12-30 00:35:21 +00:00
parsedate.h Updated the copyright year since changes have been this year. 2005-03-31 07:02:02 +00:00
progress.c As reported in Mandrake's bug tracker bug 12289 2004-11-26 14:33:13 +00:00
progress.h clean up start time and t_startsingle use so that redirect_time works properly 2004-11-15 11:27:03 +00:00
security.c Removed security.h since it shadows an include file mingw needs when building 2005-03-14 00:00:45 +00:00
select.c additional renames of Curl_ourerrno => Curl_sockerrno 2006-05-05 10:24:27 +00:00
select.h First commit of David McCreedy's EBCDIC and TPF changes. 2006-04-07 21:50:47 +00:00
sendf.c additional renames of Curl_ourerrno => Curl_sockerrno 2006-05-05 10:24:27 +00:00
sendf.h FTP code turned into state machine. Not completely yet, but a good start. 2005-02-09 13:06:40 +00:00
setup.h Added support for Salford-C under Win32 (scc). HAVE_MALLOC_H and 2006-04-26 17:11:05 +00:00
share.c Dan Fandrich added the --disable-cookies option to configure to build 2004-12-05 23:59:32 +00:00
share.h Added support for Salford-C under Win32 (scc). HAVE_MALLOC_H and 2006-04-26 17:11:05 +00:00
sockaddr.h oops * 2 2005-11-12 22:13:20 +00:00
speedcheck.c First curl_multi_socket() commit. Should primarily be considered as an internal 2006-04-10 15:00:53 +00:00
speedcheck.h updated year in the copyright string 2004-01-07 09:19:33 +00:00
splay.c First curl_multi_socket() commit. Should primarily be considered as an internal 2006-04-10 15:00:53 +00:00
splay.h First curl_multi_socket() commit. Should primarily be considered as an internal 2006-04-10 15:00:53 +00:00
sslgen.c Xavier Bouchoux made the SSL connection non-blocking for the multi interface 2006-03-21 21:54:44 +00:00
sslgen.h Xavier Bouchoux made the SSL connection non-blocking for the multi interface 2006-03-21 21:54:44 +00:00
ssluse.c Robson Braga Araujo fixed two problems in the recently added non-blocking SSL 2006-05-09 12:43:49 +00:00
ssluse.h Xavier Bouchoux made the SSL connection non-blocking for the multi interface 2006-03-21 21:54:44 +00:00
strequal.c removed tabs and trailing whitespace from source 2004-10-06 07:50:18 +00:00
strequal.h modified this year 2005-04-22 20:48:07 +00:00
strerror.c Added revision ID-tag. 2006-04-26 17:27:36 +00:00
strerror.h removed trailing whitespace 2004-10-11 17:26:24 +00:00
strtok.c Dan Fandrich's cleanup patch to make pedantic compiler options cause less 2004-01-29 13:56:45 +00:00
strtok.h updated year in the copyright string 2004-01-07 09:19:33 +00:00
strtoofft.c Made the copyright year match the latest modification's year. 2006-01-09 13:17:14 +00:00
strtoofft.h Watcom uses 'i64' suffix. 2004-12-17 18:33:09 +00:00
telnet.c Roland Blom filed bug report #1481217 2006-05-04 22:39:47 +00:00
telnet.h FTP code turned into state machine. Not completely yet, but a good start. 2005-02-09 13:06:40 +00:00
tftp.c oops, could return an uninitialized variable 2006-05-09 13:02:53 +00:00
tftp.h John Kelly added TFTP support to libcurl. A bunch of new error codes was 2005-09-02 15:11:08 +00:00
timeval.c Made the copyright year match the latest modification's year. 2006-01-09 13:17:14 +00:00
timeval.h Made the copyright year match the latest modification's year. 2006-01-09 13:17:14 +00:00
transfer.c David McCreedy brought line end conversions when doing FTP ASCII 2006-04-26 07:40:37 +00:00
transfer.h First curl_multi_socket() commit. Should primarily be considered as an internal 2006-04-10 15:00:53 +00:00
url.c First curl_multi_socket() commit. Should primarily be considered as an internal 2006-04-10 15:00:53 +00:00
url.h First curl_multi_socket() commit. Should primarily be considered as an internal 2006-04-10 15:00:53 +00:00
urldata.h Fix GnuTLS compile warning. Risking breakage with some older version of GnuTLS? 2006-05-07 18:27:36 +00:00
version.c CURL_VERSION_CONV is returned by curl_version_info if libcurl has been built 2006-04-19 09:03:21 +00:00

README.multi_socket

Implementation of the curl_multi_socket API

  Most of the design decisions and debates about this new API have already
  been held on the curl-library mailing list a long time ago so I had a basic
  idea on what approach to use. The main ideas of the new API are simply:

   1 - The application can use whatever event system it likes as it gets info
       from libcurl about what file descriptors libcurl waits for what action
       on. (The previous API returns fd_sets which is very select()-centric).

   2 - When the application discovers action on a single socket, it calls
       libcurl and informs that there was action on this particular socket and
       libcurl can then act on that socket/transfer only and not care about
       any other transfers. (The previous API always had to scan through all
       the existing transfers.)

  The idea is that curl_multi_socket() calls a given callback with information
  about what socket to wait for what action on, and the callback only gets
  called if the status of that socket has changed.

  In the API draft from before, we have a timeout argument on a per socket
  basis and we also allowed curl_multi_socket() to pass in an 'easy handle'
  instead of socket to allow libcurl to shortcut a lookup and work on the
  affected easy handle right away. Both these turned out to be bad ideas.

  The timeout argument was removed from the socket callback since after much
  thinking I came to the conclusion that we really don't want to handle
  timeouts on a per socket basis. We need it on a per transfer (easy handle)
  basis and thus we can't provide it in the callbacks in a nice way. Instead,
  we have to offer a curl_multi_timeout() that returns the largest amount of
  time we should wait before we call the "timeout action" of libcurl, to
  trigger the proper internal timeout action on the affected transfer. To get
  this to work, I added a struct to each easy handle in which we store an
  "expire time" (if any). The structs are then "splay sorted" so that we can
  add and remove times from the linked list and yet somewhat swiftly figure
  out 1 - how long time there is until the next timer expires and 2 - which
  timer (handle) should we take care of now. Of course, the upside of all this
  is that we get a curl_multi_timeout() that should also work with old-style
  applications that use curl_multi_perform().

  The easy handle argument was removed fom the curl_multi_socket() function
  because having it there would require the application to do a socket to easy
  handle conversion on its own. I find it very unlikely that applications
  would want to do that and since libcurl would need such a lookup on its own
  anyway since we didn't want to force applications to do that translation
  code (it would be optional), it seemed like an unnecessary option.

  Instead I created an internal "socket to easy handles" hash table that given
  a socket (file descriptor) return the easy handle that waits for action on
  that socket.  This hash is made using the already existing hash code
  (previously only used for the DNS cache).

  To make libcurl be able to report plain sockets in the socket callback, I
  had to re-organize the internals of the curl_multi_fdset() etc so that the
  conversion from sockets to fd_sets for that function is only done in the
  last step before the data is returned. I also had to extend c-ares to get a
  function that can return plain sockets, as that library too returned only
  fd_sets and that is no longer good enough. The changes done to c-ares have
  been committed and are available in the c-ares CVS repository destined to be
  included in the upcoming c-ares 1.3.1 release.

  The 'shiper' tool is the test application I wrote that uses the new
  curl_multi_socket() in its current state. It seems to be working and it uses
  the API as it is documented and supposed to work. It is still using
  select(), because I needed that during development (like until I had the
  socket hash implemented etc) and because I haven't yet learned how to use
  libevent or similar.

  The hiper/shiper tools are very simple and initiates lots of connections and
  have them running for the test period and then kills them all.

  Since I wasn't done with the implementation until early January I haven't
  had time to run very many measurements and checks, but I have done a few
  runs with up to a few hundred connections (with a single active one). The
  curl_multi_socket() invoke then takes 3-6 microseconds in average (using the
  read-only-1-byte-at-a-time hack). If this number does increase a lot when we
  add connections, it certainly matches my in my opinion very ambitious goal.
  We are now below the 60 microseconds "per socket action" goal. It is
  destined to be somewhat higher the more connections we have since the hash
  table gets more populated and the splay tree will grow etc.

  Some tests at 7000 and 9000 connections showed that the socket hash lookup
  is somewhat of a bottle neck. Its current implementation may be a bit too
  limiting. It simply has a fixed-size array, and on each entry in the array
  it has a linked list with entries. So the hash only checks which list to
  scan through. The code I had used so for used a list with merely 7 slots (as
  that is what the DNS hash uses) but with 7000 connections that would make an
  average of 1000 nodes in each list to run through. I upped that to 97 slots
  (I believe a prime is suitable) and noticed a significant speed increase.  I
  need to reconsider the hash implementation or use a rather large default
  value like this. At 9000 connections I was still below 10us per call.

Status Right Now

  The curl_multi_socket() API is implemented according to how it is
  documented.

    http://curl.haxx.se/libcurl/c/curl_multi_socket.html
    http://curl.haxx.se/libcurl/c/curl_multi_timeout.html
    http://curl.haxx.se/libcurl/c/curl_multi_setopt.html

What is Left for the curl_multi_socket API

  1 - More measuring with more extreme number of connections

  2 - More testing with actual URLs and complete from start to end transfers.

  I'm quite sure we don't set expire times all over in the code properly, so
  there is bound to be some timeout bugs left.

  What it really takes is for me to commit the code and to make an official
  release with it so that we get people "out there" to help out testing it.