1
0
mirror of https://github.com/moparisthebest/curl synced 2024-08-13 17:03:50 -04:00
Commit Graph

430 Commits

Author SHA1 Message Date
Steve Holme
36f7b399de multi: Prefer we don't use CURLE_OK and NULL in comparisons 2014-11-18 20:25:05 +00:00
Daniel Stenberg
c068284882 multi_runsingle: use 'result' for local CURLcode storage
... and assign data->result only at the end. Makes the code more compact
(easier to read) and more similar to other code.
2014-11-18 15:18:21 +01:00
Daniel Stenberg
e05f801271 multi_runsingle: rename result to rc
save 'result' for CURLcode types
2014-11-18 14:47:15 +01:00
Daniel Stenberg
e04ccbd506 multi: make multi_runsingle loop internally
simplifies the use of this function at little cost.
2014-11-18 14:47:15 +01:00
Carlo Wood
1342a96ecf multi: when leaving for timeout, close accordingly
Fixes the problem when a transfer in a pipeline times out.
2014-11-18 14:47:15 +01:00
Daniel Stenberg
f64dbb08c2 multi: removed Curl_multi_set_easy_connection
It isn't used anywhere!

Reported-by: Carlo Wood
2014-11-10 10:09:40 +01:00
Daniel Stenberg
0eb3d15ccb code cleanup: we prefer 'CURLcode result'
... for the local variable name in functions holding the return
code. Using the same name universally makes code easier to read and
follow.

Also, unify code for checking for CURLcode errors with:

 if(result) or if(!result)

instead of

 if(result == CURLE_OK), if(CURLE_OK == result) or if(result != CURLE_OK)
2014-10-24 08:23:19 +02:00
Daniel Stenberg
d1b56d0043 multi_runsingle: fix possible memory leak
Coverity CID 1202837. 'newurl' can in fact be allocated even when
Curl_retry_request() returns failure so free it if need be.
2014-10-07 13:57:13 +02:00
Daniel Stenberg
b0bfae1963 curl_multi_remove_handle: remove dead code
Coverify CID 1157776. Removed a superfluous if() that always evaluated
true (and an else clause that never ran), and then re-indented the
function accordingly.
2014-10-03 23:46:10 +02:00
Daniel Stenberg
0d357155cc multi_runsingle: fix memory leak
Coverity CID 1202837. There's a potential risk that 'newurl' gets
overwritten when it was already pointing to allocated memory.
2014-10-02 23:22:01 +02:00
Steve Holme
4a6fa4c204 multi.c: Avoid invalid memory read after free() from commit 3c8c873252
As the current element in the list is free()d by Curl_llist_remove(),
when the associated connection is pending, reworked the loop to avoid
accessing the next element through e->next afterward.
2014-09-07 07:11:14 +01:00
Steve Holme
c25cd9094b multi.c: Fixed compilation warning from commit 3c8c873252
warning: implicit conversion from enumeration type 'CURLMcode' to
different enumeration type 'CURLcode'
2014-09-07 00:21:36 +01:00
Daniel Stenberg
a6c48c8be7 curl_multi_cleanup: remove superfluous NULL assigns
... as the struct is free()d in the end anyway. It was first pointed out
to me that one of the ->msglist assignments were supposed to have been
->pending but was a copy and paste mistake when I realized none of the
clearing of pointers had to be there.
2014-09-02 23:44:42 +02:00
Daniel Stenberg
3c8c873252 multi: convert CURLM_STATE_CONNECT_PEND handling to a list
... instead of scanning through all handles, stash only the actual
handles that are in that state in the new ->pending list and scan that
list only. It should be mostly empty or very short. And only used for
pipelining.

This avoids a rather hefty slow-down especially notable if you add many
handles to the same multi handle. Regression introduced in commit
0f147887 (version 7.30.0).

Bug: http://curl.haxx.se/mail/lib-2014-07/0206.html
Reported-by: David Meyer
2014-09-02 10:17:47 +02:00
Daniel Stenberg
cacdc27f52 low-speed-limit: avoid timeout flood
Introducing Curl_expire_latest(). To be used when we the code flow only
wants to get called at a later time that is "no later than X" so that
something can be checked (and another timeout be added).

The low-speed logic for example could easily be made to set very many
expire timeouts if it would be called faster or sooner than what it had
set its own timer and this goes for a few other timers too that aren't
explictiy checked for timer expiration in the code.

If there's no condition the code that says if(time-passed >= TIME), then
Curl_expire_latest() is preferred to Curl_expire().

If there exists such a condition, it is on the other hand important that
Curl_expire() is used and not the other.

Bug: http://curl.haxx.se/mail/lib-2014-06/0235.html
Reported-by: Florian Weimer
2014-08-31 23:50:01 +02:00
Michael Wallner
09b5a99816 resolve: cache lookup for async resolvers
While waiting for a host resolve, check if the host cache may have
gotten the name already (by someone else), for when the same name is
resolved by several simultanoues requests.

The resolver thread occasionally gets stuck in getaddrinfo() when the
DNS or anything else is crappy or slow, so when a host is found in the
DNS cache, leave the thread alone and let itself cleanup the mess.
2014-08-31 10:49:40 +02:00
Daniel Stenberg
a4cece3d47 CONNECT: Revert Curl_proxyCONNECT back to 7.29.0 design
This reverts commit cb3e6dfa35 and instead fixes the problem
differently.

The reverted commit addressed a test failure in test 1021 by simplifying
and generalizing the code flow in a way that damaged the
performance. Now we modify the flow so that Curl_proxyCONNECT() again
does as much as possible in one go, yet still do test 1021 with and
without valgrind. It failed due to mistakes in the multi state machine.

Bug: http://curl.haxx.se/bug/view.cgi?id=1397
Reported-by: Paul Saab
2014-07-22 23:00:19 +02:00
Ray Satiro
907520c4b9 progress callback: skip last callback update on errors
When an error has been detected, skip the final forced call to the
progress callback by making sure to pass the current return code
variable in the Curl_done() call in the CURLM_STATE_DONE state.

This avoids the "extra" callback that could occur even if you returned
error from the progress callback.

Bug: http://curl.haxx.se/mail/lib-2014-06/0062.html
Reported by: Jonathan Cardoso Machado
2014-07-02 23:53:25 +02:00
Daniel Stenberg
df13f8e8c2 bits.close: introduce connection close tracking
Make all code use connclose() and connkeep() when changing the "close
state" for a connection. These two macros take a string argument with an
explanation, and debug builds of curl will include that in the debug
output. Helps tracking connection re-use/close issues.
2014-05-22 00:34:10 +02:00
Daniel Stenberg
84bd19ffd4 timers: fix timer regression involving redirects / reconnects
In commit 0b3750b5c2 (released in 7.36.0) we fixed a timeout issue
but instead broke the timings.

To fix this, I introduce a new timestamp to use for the timeouts and
restored the previous timestamp and timestamp position so that the old
timer functionality is restored.

In addition to that, that change also broke connection timeouts for when
more than one connect was used (as it would then count the total time
from the first connect and not for the most recent one). Now
Curl_timeleft() has been modified so that it checks against different
start times depending on which timeout it checks.

Test 1303 is updated accordingly.

Bug: http://curl.haxx.se/mail/lib-2014-05/0147.html
Reported-by: Ryan Braud
2014-05-15 21:28:19 +02:00
Jeff King
37f484276d curl_multi_cleanup: ignore SIGPIPE better
When looping and closing each individual connection left open, the
SIGPIPE ignoring was not done and could thus lead to death by signal 13.

Bug: http://thread.gmane.org/gmane.comp.version-control.git/238242
2014-05-05 12:47:46 +02:00
Daniel Stenberg
d5ec44ca4c INFILESIZE: fields in UserDefined must not be changed run-time
set.infilesize in this case was modified in several places, which could
lead to repeated requests using the same handle to get unintendent/wrong
consequences based on what the previous request did!
2014-04-26 18:17:10 +02:00
Marc Hoersken
ee6791128f multi.c: fix possible invalid memory access in case nfds overflows
ufds might not be allocated in case nfds overflows to zero while
extra_nfds is still non-zero. udfs is then accessed within the
extra_nfds-based for loop.
2014-04-19 16:02:14 +02:00
Daniel Stenberg
0b3750b5c2 multi_runsingle: move timestamp into INIT
Setting the TIMER_STARTSINGLE timestamp first in CONNECT has the
drawback that for actions that go back to the CONNECT state, the time
stamp is reset and for the multi_socket API there's no corresponding
Curl_expire() then so the timeout logic gets wrong!

Reported-by: Brad Spencer
Bug: http://curl.haxx.se/mail/lib-2014-02/0036.html
2014-03-03 16:35:04 +01:00
Daniel Stenberg
854aca5420 multi: ignore sigpipe internally
When the multi API is used we must also ignore SIGPIPE signals when
caused by things we do, like they can easily be generated by OpenSSL.
2014-02-17 10:35:21 +01:00
Shao Shuchao
2111c2ed07 ConnectionDone: default maxconnects to 4 x number of easy handles
... as documented!
2014-02-17 09:29:25 +01:00
Dan Fandrich
c631a54bb6 ftp: fixed a memory leak on wildcard error path 2014-01-30 22:15:03 +01:00
Dan Fandrich
da9864fb72 multi: Fixed a memory leak on OOM condition 2014-01-24 00:22:08 +01:00
Colin Hogben
bbc1705fa9 error message: Sensible message on timeout when transfer size unknown
A transfer timeout could result in an error message such as "Operation
timed out after 3000 milliseconds with 19 bytes of -1 received".  This
patch removes the non-sensical "of -1" when the size of the transfer
is unknown, mirroring the logic in lib/transfer.c
2014-01-13 16:54:10 +01:00
Daniel Stenberg
3b183df9cc multi: remove MULTI_TIMEOUT_INACCURACY
With the recently added timeout "reminder" functionality, there's no
reason left for us to execute timeout code before the time is
ripe. Simplifies the handling too.

This will make the *TIMEOUT and *CONNECTTIMEOUT options more accurate
again, which probably is most important when the *_MS versions are used.

In multi_socket, make sure to update 'now' after having handled activity
on a socket.
2014-01-12 00:11:53 +01:00
Daniel Stenberg
980659a2ca multi_socket: remind app if timeout didn't run
BACKGROUND:

We have learned that on some systems timeout timers are inaccurate and
might occasionally fire off too early. To make the multi_socket API work
with this, we made libcurl execute timeout actions a bit early too if
they are within our MULTI_TIMEOUT_INACCURACY. (added in commit
2c72732ebf, present since 7.21.0)

Switching everything to the multi API made this inaccuracy problem
slightly more notable as now everyone can be affected.

Recently (commit 21091549c0) we tweaked that inaccuracy value to make
timeouts more accurate and made it platform specific. We also figured
out that we have code at places that check for fixed timeout values so
they MUST NOT run too early as then they will not trigger at all (see
commit be28223f35 and a691e04470) - so there are definitately problems
with running timeouts before they're supposed to run. (We've handled
that so far by adding the inaccuracy margin to those specific timeouts.)

The libcurl multi_socket API tells the application with a callback that
a timeout expires in N milliseconds (and it explicitly will not tell it
again for the same timeout), and the application is then supposed to
call libcurl when that timeout expires. When libcurl subsequently gets
called with curl_multi_socket_action(...CURL_SOCKET_TIMEOUT...), it
knows that the application thinks the timeout expired - and alas, if it
is within the inaccuracy level libcurl will run code handling that
handle.

If the application says CURL_SOCKET_TIMEOUT to libcurl and _isn't_
within the inaccuracy level, libcurl will not consider the timeout
expired and it will not tell the application again since the timeout
value is still the same.

NOW:

This change introduces a modified behavior here. If the application says
CURL_SOCKET_TIMEOUT and libcurl finds no timeout code to run, it will
inform the application about the timeout value - *again* even if it is
the same timeout that it already told about before (although libcurl
will of course tell it the updated time so that it'll still get the
correct remaining time). This way, we will not risk that the application
believes it has done its job and libcurl thinks the time hasn't come yet
to run any code and both just sit waiting. This also allows us to
decrease the MULTI_TIMEOUT_INACCURACY margin, but that will be handled
in a separate commit.

A repeated timeout update to the application risk that the timeout will
then fire again immediately and we have what basically is a busy-loop
until the time is fine even for libcurl. If that becomes a problem, we
need to address it.
2014-01-10 13:57:25 +01:00
Marc Hoersken
61312fe66f multi.c: fix possible dereference of null pointer 2014-01-05 20:01:33 +01:00
Daniel Stenberg
bf24b64e83 progresscallback: make CURLE_ABORTED_BY_CALLBACK get returned better
When the progress callback returned 1 at a very early state, the code
would not make CURLE_ABORTED_BY_CALLBACK get returned but the process
would still be interrupted. In the HTTP case, this would then cause a
CURLE_GOT_NOTHING to erroneously get returned instead.

Reported-by: Petr Novak
Bug: http://curl.haxx.se/bug/view.cgi?id=1318
2014-01-03 14:09:59 +01:00
Steve Holme
60bd22620a mprintf: Replaced internal usage of FORMAT_OFF_T and FORMAT_OFF_TU
Following commit 0aafd77fa4, replaced the internal usage of
FORMAT_OFF_T and FORMAT_OFF_TU with the external versions that we
expect API programmers to use.

This negates the need for separate definitions which were subtly
different under different platforms/compilers.
2013-12-31 11:10:42 +00:00
Daniel Stenberg
2715d7f948 FILE: don't wait due to CURLOPT_MAX_RECV_SPEED_LARGE
The FILE:// code doesn't support this option - and it doesn't make sense
to support it as long as it works as it does since then it'd only block
even longer.

But: setting CURLOPT_MAX_RECV_SPEED_LARGE would make the transfer first
get done and then libcurl would wait until the average speed would get
low enough. This happened because the transfer happens completely in the
DO state for FILE:// but then it would still unconditionally continue in
to the PERFORM state where the speed check is made.

Starting now, the code will skip from DO_DONE to DONE immediately if no
socket is set to be recv()ed or send()ed to.

Bug: http://curl.haxx.se/bug/view.cgi?id=1312
Reported-by: Mohammad AlSaleh
2013-12-22 23:44:14 +01:00
Steve Holme
d92de3a7e9 multi.c: Fixed compilation warning
warning: declaration of 'pipe' shadows a global declaration
2013-12-01 20:22:57 +00:00
Steve Holme
26ff1ea6c3 multi.c: Fixed compilation error introduced in commit a900d45489
Systems that define SIGPIPE_VARIABLE as a noop would not compile as
restore_pipe was defined afterwards.
2013-11-27 23:45:45 +00:00
Jeff King
a900d45489 curl_multi_cleanup: ignore SIGPIPE
This is an extension to the fix in 7d80ed64e4. We may
call Curl_disconnect() while cleaning up the multi handle,
which could lead to openssl sending packets, which could get
a SIGPIPE.

Signed-off-by: Jeff King <peff@peff.net>
2013-11-27 22:47:12 +01:00
Steve Holme
65ce9b6d6c multi: Small code tidy up to avoid hard return 2013-11-12 09:59:22 +00:00
Björn Stenberg
fa1253aee1 multi: Set read socket when returning READSOCK(0)
This patch fixes and issue introduced in commit 7d7df83198, if the
tunnel state was TUNNEL_CONNECT, waitconnect_getsock() would return a
bitmask indicating a readable socket but never stored the socket in the
return array.
2013-11-11 23:41:44 +00:00
Björn Stenberg
02fbc26d59 connect: Add connection delay to Happy Eyeballs.
This patch adds a 200ms delay between the first and second address
family socket connection attempts.

It also iterates over IP addresses in the order returned by the
system, meaning most dual-stack systems will try IPv6 first.

Additionally, it refactors the connect code, removing most code that
handled synchronous connects. Since all sockets are now non-blocking,
the logic can be made simpler.
2013-11-04 22:38:19 +01:00
Daniel Stenberg
753d44fa00 curl_multi_wait: accept 0 from multi_timeout() as valid timeout
The code rejected 0 as a valid timeout while in fact the function could
indeed legitimately return that and it should be respected.

Reported-by: Bjorn Stenberg
2013-10-30 23:48:08 +01:00
Björn Stenberg
7d7df83198 Add "Happy Eyeballs" for IPv4/IPv6.
This patch invokes two socket connect()s nearly simultaneously, and
the socket that is first connected "wins" and is subsequently used for
the connection. The other is terminated.

There is a very slight IPv4 preference, in that if both sockets connect
simultaneously IPv4 is checked first and thus will win.
2013-10-27 11:19:53 +01:00
Daniel Stenberg
84f3b3dd44 curl_multi_remove_handle: allow multiple removes
When removing an already removed handle, avoid that to ruin the
internals and just return OK instead.
2013-08-29 22:08:45 +02:00
Daniel Stenberg
a691e04470 multi_socket: improved 100-continue timeout handling
When waiting for a 100-continue response from the server, the
Curl_readwrite() will refuse to run if called until the timeout has been
reached.

We timeout code in multi_socket() allows code to run slightly before the
actual timeout time, so for test 154 it could lead to the function being
executed but refused in Curl_readwrite() and then the application would
just sit idling forever.

This was detected with runtests.pl -e on test 154.
2013-08-28 00:07:12 +02:00
Daniel Stenberg
22adb46a32 multi: move on from STATE_DONE faster
Make sure we always return CURLM_CALL_MULTI_PERFORM when we reach
CURLM_STATE_DONE since the state is transient and it can very well
continue executing as there is nothing to wait for.

Bug: http://curl.haxx.se/mail/lib-2013-08/0211.html
Reported-by: Yi Huang
2013-08-22 22:55:25 +02:00
Daniel Stenberg
19122c0768 CURLM_ADDED_ALREADY: new error code
Doing curl_multi_add_handle() on an easy handle that is already added to
a multi handle now returns this error code. It previously returned
CURLM_BAD_EASY_HANDLE for this condition.
2013-08-20 23:13:19 +02:00
Daniel Stenberg
c346c4c8f9 multi_init: moved init code here from add_handle
The closure_handle is "owned" by the multi handle and it is
unconditional so the setting up of it should be in the Curl_multi_handle
function rather than curl_multi_add_handle.
2013-08-20 23:13:19 +02:00
Daniel Stenberg
bc7d806e3a multi: remove dns cache creation code from *add_handle
As it is done unconditionally in multi_init() this code will never run!
2013-08-20 23:13:19 +02:00
Daniel Stenberg
2ae3d28f3d multi: s/easy/data
With everything being struct SessionHandle pointers now, this rename
makes multi.c use the library-wide practise of calling that pointer
'data' instead of the previously used 'easy'.
2013-08-12 13:52:58 +02:00
Daniel Stenberg
8a42c2ef8d cleanup: removed one function, made one static
Moved Curl_easy_addmulti() from easy.c to multi.c, renamed it to
easy_addmulti and made it static.

Removed Curl_easy_initHandleData() and uses of it since it was emptied
in commit cdda92ab67b47d74a.
2013-08-12 13:17:57 +02:00
Daniel Stenberg
2af0b10c95 comments: remove old and wrong multi/easy interface statements 2013-08-09 23:39:09 +02:00
Daniel Stenberg
d2b36e466a global dns cache: didn't work [regression]
CURLOPT_DNS_USE_GLOBAL_CACHE broke in commit c43127414d (been
broken since the libcurl 7.29.0 release). While this option has been
documented as deprecated for almost a decade and nobody even reported
this bug, it should remain functional.

Added test case 1512 to verify
2013-08-08 16:28:46 +02:00
Daniel Stenberg
7cc00d9a83 FTP: when EPSV gets a 229 but fails to connect, retry with PASV
This is a regression as this logic used to work. It isn't clear when it
broke, but I'm assuming in 7.28.0 when we went all-multi internally.

This likely never worked with the multi interface. As the failed
connection is detected once the multi state has reached DO_MORE, the
Curl_do_more() function was now expanded somewhat so that the
ftp_do_more() function can request to go "back" to the previous state
when it makes another attempt - using PASV.

Added test case 1233 to verify this fix. It has the little issue that it
assumes no service is listening/accepting connections on port 1...

Reported-by: byte_bucket in the #curl IRC channel
2013-08-06 09:57:59 +02:00
Daniel Stenberg
eb41e8eebe multi: remove the one_easy struct field
Since the merge of SessionHandle with Curl_one_easy, this indirection
isn't used anymore.
2013-08-03 22:51:35 +02:00
Daniel Stenberg
3cd43bbfec multi: rename all Curl_one_easy to SessionHandle 2013-08-03 22:51:35 +02:00
Daniel Stenberg
204e340bcd multi: remove the multi_pos struct field
Since Curl_one_easy is really a SessionHandle now, this indirection
doesn't exist anymore.
2013-08-03 22:51:35 +02:00
Daniel Stenberg
37f2ba7e57 multi: remove easy_handle struct field
It isn't needed anymore
2013-08-03 22:51:35 +02:00
Daniel Stenberg
09b9fc9009 multi: remove 'Curl_one_easy' struct, phase 1
The motivation for having a separate struct that keep track of an easy
handle when using the multi handle was removed when we switched to
always using the multi interface internally. Now they were just two
separate struct that was always allocated for each easy handle.

This first step just moves the Curl_one_easy struct members into the
SessionHandle struct and hides this somehow (== keeps the source code
changes to a minimum) by defining Curl_one_easy to SessionHandle

The biggest changes in this commit are:

 1 - the linked list of easy handles had to be changed somewhat due
     to the new struct layout. This made the main linked list pointer
     get renamed to 'easyp' and there's also a new pointer to the last
     node, called easylp. It is no longer circular but ends with ->next
     pointing to NULL. New nodes are still added last.

 2 - easy->state is now called easy->mstate to avoid name collision
2013-08-03 22:51:35 +02:00
Yang Tse
4fad1943a2 string formatting: fix 15+ printf-style format strings 2013-07-24 16:46:24 +02:00
Yang Tse
de052ca6fc string formatting: fix 25+ printf-style format strings 2013-07-24 01:21:26 +02:00
Daniel Stenberg
513e587c5e curl_multi_wait: fix revents
Commit 6d30f8ebed didn't work properly. First, it used the wrong
array index, but this fix also:

1 - only does the copying if indeed there was any activity

2 - makes sure to properly translate between internal and external
bitfields, which are not guaranteed to match

Reported-by: Evgeny Turnaev
2013-07-21 21:54:47 +02:00
Evgeny Turnaev
6d30f8ebed curl_multi_wait: set revents for extra fds
Pass back the revents that happened for the user-provided file
descriptors.
2013-07-18 00:06:09 +02:00
Daniel Stenberg
88c5c63ffc multi_socket: react on socket close immediately
As a remedy to the problem when a socket gets closed and a new one is
opened with the same file descriptor number and as a result
multi.c:singlesocket() doesn't detect the difference, the new function
Curl_multi_closed() gets told when a socket is closed so that it can be
removed from the socket hash. When the old one has been removed, a new
socket should be detected fine by the singlesocket() on next invoke.

Bug: http://curl.haxx.se/bug/view.cgi?id=1248
Reported-by: Erik Johansson
2013-06-20 22:36:52 +02:00
Daniel Stenberg
21091549c0 multi_socket: reduce timeout inaccuracy margin
Allow less room for "triggered too early" mistakes by applications /
timers on non-windows platforms. Starting now, we assume that a timeout
call is never made earlier than 3 milliseconds before the actual
timeout. This greatly improves timeout accuracy on Linux.

Bug: http://curl.haxx.se/bug/view.cgi?id=1228
Reported-by: Hang Su
2013-06-11 00:02:29 +02:00
Daniel Stenberg
0bf5ce77aa curl_multi_wait: only use internal timer if not -1
commit 29bf0598aa introduced a problem when the "internal" timeout is
prefered to the given if shorter, as it didn't consider the case where
-1 was returned. Now the internal timeout is only considered if not -1.

Reported-by: Tor Arntsen
Bug: http://curl.haxx.se/mail/lib-2013-06/0015.html
2013-06-04 13:22:40 +02:00
Daniel Stenberg
29bf0598aa curl_multi_wait: reduce timeout if the multi handle wants to
If the multi handle's pending timeout is less than what is passed into
this function, it will now opt to use the shorter time anyway since it
is a very good hint that the handle wants to process something in a
shorter time than what otherwise would happen.

curl_multi_wait.3 was updated accordingly to clarify

This is the reason for bug #1224

Bug: http://curl.haxx.se/bug/view.cgi?id=1224
Reported-by: Andrii Moiseiev
2013-06-03 20:27:08 +02:00
Daniel Stenberg
239b58d34d multi_runsingle: switch an if() condition for readability
... because there's an identical check right next to it so using the
operators in the check in the same order increases readability.
2013-06-03 20:23:01 +02:00
Daniel Stenberg
6691fdf517 multi_runsingle: add braces to clarify the code 2013-05-30 23:34:33 +02:00
Daniel Stenberg
6420672879 curl_easy_init: use less mallocs
By introducing an internal alternative to curl_multi_init() that accepts
parameters to set the hash sizes, easy handles will now use tiny socket
and connection hash tables since it will only ever add a single easy
handle to that multi handle.

This decreased the number mallocs in test 40 (which is a rather simple
and typical easy interface use case) from 1142 to 138. The maximum
amount of memory allocated used went down from 118969 to 78805.
2013-04-26 22:55:55 +02:00
Steve Holme
d85647cfdd multi.c: Corrected a couple of violations of the curl coding standards
Corrected some incorrectly positioned pointer variable declarations to
be "type *" rather than "type* ".
2013-03-21 19:14:03 +00:00
Steve Holme
7713e67bc5 multi.c: Fix compilation warning
warning: an enumerated type is mixed with another type
2013-03-21 07:33:45 +00:00
Steve Holme
9a13a516b4 multi.c: fix compilation error
warning: conversion from enumeration type to different enumeration type
2013-03-20 23:36:46 +00:00
Linus Nielsen Feltzing
0f147887b0 Multiple pipelines and limiting the number of connections.
Introducing a number of options to the multi interface that
allows for multiple pipelines to the same host, in order to
optimize the balance between the penalty for opening new
connections and the potential pipelining latency.

Two new options for limiting the number of connections:

CURLMOPT_MAX_HOST_CONNECTIONS - Limits the number of running connections
to the same host. When adding a handle that exceeds this limit,
that handle will be put in a pending state until another handle is
finished, so we can reuse the connection.

CURLMOPT_MAX_TOTAL_CONNECTIONS - Limits the number of connections in total.
When adding a handle that exceeds this limit,
that handle will be put in a pending state until another handle is
finished. The free connection will then be reused, if possible, or
closed if the pending handle can't reuse it.

Several new options for pipelining:

CURLMOPT_MAX_PIPELINE_LENGTH - Limits the pipeling length. If a
pipeline is "full" when a connection is to be reused, a new connection
will be opened if the CURLMOPT_MAX_xxx_CONNECTIONS limits allow it.
If not, the handle will be put in a pending state until a connection is
ready (either free or a pipe got shorter).

CURLMOPT_CONTENT_LENGTH_PENALTY_SIZE - A pipelined connection will not
be reused if it is currently processing a transfer with a content
length that is larger than this.

CURLMOPT_CHUNK_LENGTH_PENALTY_SIZE - A pipelined connection will not
be reused if it is currently processing a chunk larger than this.

CURLMOPT_PIPELINING_SITE_BL - A blacklist of hosts that don't allow
pipelining.

CURLMOPT_PIPELINING_SERVER_BL - A blacklist of server types that don't allow
pipelining.

See the curl_multi_setopt() man page for details.
2013-03-13 23:55:24 +01:00
Daniel Stenberg
136a3a0ee2 curl_multi_wait: avoid second loop if nothing to do
... hopefully this will also make clang-analyzer stop warning on
potentional NULL dereferences (which were false positives anyway).
2013-03-09 22:27:15 +01:00
Daniel Stenberg
64b2d2d77e multi_runsingle: avoid NULL dereference
When Curl_do() returns failure, the connection pointer could be NULL so
the code path following needs to that that into account.

Bug: http://curl.haxx.se/mail/lib-2013-03/0062.html
Reported by: Eric Hu
2013-03-09 22:27:15 +01:00
Linus Nielsen Feltzing
da3fc1ee91 Fix NULL pointer reference when closing an unused multi handle. 2013-02-10 22:57:58 +01:00
Daniel Stenberg
c43127414d always-multi: always use non-blocking internals
Remove internal separated behavior of the easy vs multi intercace.
curl_easy_perform() is now using the multi interface itself.

Several minor multi interface quirks and bugs have been fixed in the
process.

Much help with debugging this has been provided by: Yang Tse
2013-01-17 19:40:35 +01:00
Yang Tse
5a053ffe80 build: fix circular header inclusion with other packages
This commit renames lib/setup.h to lib/curl_setup.h and
renames lib/setup_once.h to lib/curl_setup_once.h.

Removes the need and usage of a header inclusion guard foreign
to libcurl. [1]

Removes the need and presence of an alarming notice we carried
in old setup_once.h [2]

----------------------------------------

1 - lib/setup_once.h used __SETUP_ONCE_H macro as header inclusion guard
    up to commit ec691ca3 which changed this to HEADER_CURL_SETUP_ONCE_H,
    this single inclusion guard is enough to ensure that inclusion of
    lib/setup_once.h done from lib/setup.h is only done once.

    Additionally lib/setup.h has always used __SETUP_ONCE_H macro to
    protect inclusion of setup_once.h even after commit ec691ca3, this
    was to avoid a circular header inclusion triggered when building a
    c-ares enabled version with c-ares sources available which also has
    a setup_once.h header. Commit ec691ca3 exposes the real nature of
    __SETUP_ONCE_H usage in lib/setup.h, it is a header inclusion guard
    foreign to libcurl belonging to c-ares's setup_once.h

    The renaming this commit does, fixes the circular header inclusion,
    and as such removes the need and usage of a header inclusion guard
    foreign to libcurl. Macro __SETUP_ONCE_H no longer used in libcurl.

2 - Due to the circular interdependency of old lib/setup_once.h and the
    c-ares setup_once.h header, old file lib/setup_once.h has carried
    back from 2006 up to now days an alarming and prominent notice about
    the need of keeping libcurl's and c-ares's setup_once.h in sync.

    Given that this commit fixes the circular interdependency, the need
    and presence of mentioned notice is removed.

    All mentioned interdependencies come back from now old days when
    the c-ares project lived inside a curl subdirectory. This commit
    removes last traces of such fact.
2013-01-09 00:49:50 +01:00
Yang Tse
4a5aa6682d Revert changes relative to lib/*.[ch] recent renaming
This reverts renaming and usage of lib/*.h header files done
28-12-2012, reverting 2 commits:

  f871de0... build: make use of 76 lib/*.h renamed files
  ffd8e12... build: rename 76 lib/*.h files

This also reverts removal of redundant include guard (redundant thanks
to changes in above commits) done 2-12-2013, reverting 1 commit:

  c087374... curl_setup.h: remove redundant include guard

This also reverts renaming and usage of lib/*.c source files done
3-12-2013, reverting 3 commits:

  13606bb... build: make use of 93 lib/*.c renamed files
  5b6e792... build: rename 93 lib/*.c files
  7d83dff... build: commit 13606bbfde follow-up 1

Start of related discussion thread:

  http://curl.haxx.se/mail/lib-2013-01/0012.html

Asking for confirmation on pushing this revertion commit:

  http://curl.haxx.se/mail/lib-2013-01/0048.html

Confirmation summary:

  http://curl.haxx.se/mail/lib-2013-01/0079.html

NOTICE: The list of 2 files that have been modified by other
intermixed commits, while renamed, and also by at least one
of the 6 commits this one reverts follows below. These 2 files
will exhibit a hole in history unless git's '--follow' option
is used when viewing logs.

  lib/curl_imap.h
  lib/curl_smtp.h
2013-01-06 18:20:27 +01:00
Yang Tse
5b6e7927c6 build: rename 93 lib/*.c files
93 lib/*.c source files renamed to use our standard naming scheme.

This commit only does the file renaming.

----------------------------------------

  renamed:    lib/amigaos.c -> lib/curl_amigaos.c
  renamed:    lib/asyn-ares.c -> lib/curl_asyn_ares.c
  renamed:    lib/asyn-thread.c -> lib/curl_asyn_thread.c
  renamed:    lib/axtls.c -> lib/curl_axtls.c
  renamed:    lib/base64.c -> lib/curl_base64.c
  renamed:    lib/bundles.c -> lib/curl_bundles.c
  renamed:    lib/conncache.c -> lib/curl_conncache.c
  renamed:    lib/connect.c -> lib/curl_connect.c
  renamed:    lib/content_encoding.c -> lib/curl_content_encoding.c
  renamed:    lib/cookie.c -> lib/curl_cookie.c
  renamed:    lib/cyassl.c -> lib/curl_cyassl.c
  renamed:    lib/dict.c -> lib/curl_dict.c
  renamed:    lib/easy.c -> lib/curl_easy.c
  renamed:    lib/escape.c -> lib/curl_escape.c
  renamed:    lib/file.c -> lib/curl_file.c
  renamed:    lib/fileinfo.c -> lib/curl_fileinfo.c
  renamed:    lib/formdata.c -> lib/curl_formdata.c
  renamed:    lib/ftp.c -> lib/curl_ftp.c
  renamed:    lib/ftplistparser.c -> lib/curl_ftplistparser.c
  renamed:    lib/getenv.c -> lib/curl_getenv.c
  renamed:    lib/getinfo.c -> lib/curl_getinfo.c
  renamed:    lib/gopher.c -> lib/curl_gopher.c
  renamed:    lib/gtls.c -> lib/curl_gtls.c
  renamed:    lib/hash.c -> lib/curl_hash.c
  renamed:    lib/hmac.c -> lib/curl_hmac.c
  renamed:    lib/hostasyn.c -> lib/curl_hostasyn.c
  renamed:    lib/hostcheck.c -> lib/curl_hostcheck.c
  renamed:    lib/hostip.c -> lib/curl_hostip.c
  renamed:    lib/hostip4.c -> lib/curl_hostip4.c
  renamed:    lib/hostip6.c -> lib/curl_hostip6.c
  renamed:    lib/hostsyn.c -> lib/curl_hostsyn.c
  renamed:    lib/http.c -> lib/curl_http.c
  renamed:    lib/http_chunks.c -> lib/curl_http_chunks.c
  renamed:    lib/http_digest.c -> lib/curl_http_digest.c
  renamed:    lib/http_negotiate.c -> lib/curl_http_negotiate.c
  renamed:    lib/http_negotiate_sspi.c -> lib/curl_http_negotiate_sspi.c
  renamed:    lib/http_proxy.c -> lib/curl_http_proxy.c
  renamed:    lib/idn_win32.c -> lib/curl_idn_win32.c
  renamed:    lib/if2ip.c -> lib/curl_if2ip.c
  renamed:    lib/imap.c -> lib/curl_imap.c
  renamed:    lib/inet_ntop.c -> lib/curl_inet_ntop.c
  renamed:    lib/inet_pton.c -> lib/curl_inet_pton.c
  renamed:    lib/krb4.c -> lib/curl_krb4.c
  renamed:    lib/krb5.c -> lib/curl_krb5.c
  renamed:    lib/ldap.c -> lib/curl_ldap.c
  renamed:    lib/llist.c -> lib/curl_llist.c
  renamed:    lib/md4.c -> lib/curl_md4.c
  renamed:    lib/md5.c -> lib/curl_md5.c
  renamed:    lib/memdebug.c -> lib/curl_memdebug.c
  renamed:    lib/mprintf.c -> lib/curl_mprintf.c
  renamed:    lib/multi.c -> lib/curl_multi.c
  renamed:    lib/netrc.c -> lib/curl_netrc.c
  renamed:    lib/non-ascii.c -> lib/curl_non_ascii.c
  renamed:    lib/curl_non-ascii.h -> lib/curl_non_ascii.h
  renamed:    lib/nonblock.c -> lib/curl_nonblock.c
  renamed:    lib/nss.c -> lib/curl_nss.c
  renamed:    lib/nwlib.c -> lib/curl_nwlib.c
  renamed:    lib/nwos.c -> lib/curl_nwos.c
  renamed:    lib/openldap.c -> lib/curl_openldap.c
  renamed:    lib/parsedate.c -> lib/curl_parsedate.c
  renamed:    lib/pingpong.c -> lib/curl_pingpong.c
  renamed:    lib/polarssl.c -> lib/curl_polarssl.c
  renamed:    lib/pop3.c -> lib/curl_pop3.c
  renamed:    lib/progress.c -> lib/curl_progress.c
  renamed:    lib/qssl.c -> lib/curl_qssl.c
  renamed:    lib/rawstr.c -> lib/curl_rawstr.c
  renamed:    lib/rtsp.c -> lib/curl_rtsp.c
  renamed:    lib/security.c -> lib/curl_security.c
  renamed:    lib/select.c -> lib/curl_select.c
  renamed:    lib/sendf.c -> lib/curl_sendf.c
  renamed:    lib/share.c -> lib/curl_share.c
  renamed:    lib/slist.c -> lib/curl_slist.c
  renamed:    lib/smtp.c -> lib/curl_smtp.c
  renamed:    lib/socks.c -> lib/curl_socks.c
  renamed:    lib/socks_gssapi.c -> lib/curl_socks_gssapi.c
  renamed:    lib/socks_sspi.c -> lib/curl_socks_sspi.c
  renamed:    lib/speedcheck.c -> lib/curl_speedcheck.c
  renamed:    lib/splay.c -> lib/curl_splay.c
  renamed:    lib/ssh.c -> lib/curl_ssh.c
  renamed:    lib/sslgen.c -> lib/curl_sslgen.c
  renamed:    lib/ssluse.c -> lib/curl_ssluse.c
  renamed:    lib/strdup.c -> lib/curl_strdup.c
  renamed:    lib/strequal.c -> lib/curl_strequal.c
  renamed:    lib/strerror.c -> lib/curl_strerror.c
  renamed:    lib/strtok.c -> lib/curl_strtok.c
  renamed:    lib/strtoofft.c -> lib/curl_strtoofft.c
  renamed:    lib/telnet.c -> lib/curl_telnet.c
  renamed:    lib/tftp.c -> lib/curl_tftp.c
  renamed:    lib/timeval.c -> lib/curl_timeval.c
  renamed:    lib/transfer.c -> lib/curl_transfer.c
  renamed:    lib/url.c -> lib/curl_url.c
  renamed:    lib/version.c -> lib/curl_version.c
  renamed:    lib/warnless.c -> lib/curl_warnless.c
  renamed:    lib/wildcard.c -> lib/curl_wildcard.c

----------------------------------------
2013-01-03 06:13:18 +01:00
Yang Tse
13606bbfde build: make use of 93 lib/*.c renamed files
93 *.c source files renamed to use our standard naming scheme.

This change affects 77 files in libcurl's source tree.
2013-01-03 05:50:26 +01:00
Yang Tse
f871de0064 build: make use of 76 lib/*.h renamed files
76 private header files renamed to use our standard naming scheme.

This change affects 322 files in libcurl's source tree.
2012-12-28 19:37:11 +01:00
Yang Tse
219fe7b29c curl_multi_remove_handle: commit 0aabfd9963 follow-up 2012-12-26 06:01:54 +01:00
Yang Tse
0aabfd9963 curl_multi_remove_handle: fix memory leak triggered with CURLOPT_RESOLVE 2012-12-25 13:34:45 +01:00
Yang Tse
7a09907146 curl_multi_wait: OOM handling fix 2012-12-23 21:50:39 +01:00
Daniel Stenberg
a1fc9b80c8 curl_multi_wait: avoid an unnecessary memory allocation 2012-12-23 21:50:14 +01:00
Yang Tse
b3d91a147f multi.c: OOM handling fix 2012-12-21 19:48:07 +01:00
Yang Tse
eafccdb315 bundles connection caching: some out of memory handling fixes 2012-12-19 19:53:17 +01:00
Yang Tse
a0b207164c setup_once.h: refactor inclusion of <unistd.h> and <sys/socket.h>
Inclusion of top two most included header files now done in setup_once.h
2012-12-14 17:38:18 +01:00
Linus Nielsen Feltzing
d021f2e8a0 Introducing a new persistent connection caching system using "bundles".
A bundle is a list of all persistent connections to the same host.
The connection cache consists of a hash of bundles, with the
hostname as the key.
The benefits may not be obvious, but they are two:

1) Faster search for connections to reuse, since the hash
   lookup only finds connections to the host in question.
2) It lays out the groundworks for an upcoming patch,
   which will introduce multiple HTTP pipelines.

This patch also removes the awkward list of "closure handles",
which were needed to send QUIT commands to the FTP server
when closing a connection.
Now we allocate a separate closure handle and use that
one to close all connections.

This has been tested in a live system for a few weeks, and of
course passes the test suite.
2012-12-07 10:08:33 +01:00
Daniel Stenberg
8b02afd9a9 multi: fix re-sending request on early connection close
This handling already works with the easy-interface code. When a request
is sent on a re-used connection that gets closed by the server at the
same time as the request is sent, the situation may occur so that we can
send the request and we discover the broken connection as a RECV_ERROR
in the PERFORM state and then the request needs to be retried on a fresh
connection. Test 64 broke with 'multi-always-internally'.
2012-12-04 22:14:23 +01:00
Yang Tse
b33074d893 multi.c: disambiguate precedence of bitwise and relational operation 2012-11-26 16:23:47 +01:00
Daniel Stenberg
409f2a041f fixed memory leak: CURLOPT_RESOLVE with multi interface
DNS cache entries populated with CURLOPT_RESOLVE were not properly freed
again when done using the multi interface.

Test case 1502 added to verify.

Bug: http://curl.haxx.se/bug/view.cgi?id=3575448
Reported by: Alex Gruz
2012-11-18 16:39:31 +01:00
Daniel Stenberg
8373ca3641 curl_multi_wait: no wait if no descriptors to wait for
This is a minor change in behavior after having been pointed out by Mark
Tully and discussed on the list. Initially this case would internally
call poll() with no sockets and a timeout which would equal a sleep for
that specified time.

Bug: http://curl.haxx.se/mail/lib-2012-10/0076.html
Reported by: Mark Tully
2012-10-09 22:19:49 +02:00
Daniel Stenberg
971f5bcedd multi_runsingle: CURLOPT_LOW_SPEED_* fix for rate limitation
During the periods of rate limitation, the speedcheck function wasn't
called and thus the values weren't updated accordingly and it would then
easily trigger wrongly once data got transferred again.

Also, the progress callback's return code was not acknowledged in this
state so it could make an "abort" return code to get ignored and not
have the documented effect of aborting an ongoing transfer.

Bug: http://curl.haxx.se/mail/lib-2012-09/0081.html
Reported by: Jie He
2012-10-02 00:16:20 +02:00
Sara Golemon
b78944146a curl_multi_wait: Add parameter to return number of active sockets
Minor change to recently introduced function.  BC breaking, but since
curl_multi_wait() doesn't exist in any releases that should be fine.
2012-09-16 19:58:02 +02:00
Sara Golemon
de24d7bd4c multi: add curl_multi_wait()
/*
 * Name:     curl_multi_wait()
 *
 * Desc:     Poll on all fds within a CURLM set as well as any
 *           additional fds passed to the function.
 *
 * Returns:  CURLMcode type, general multi error code.
 */
CURL_EXTERN CURLMcode curl_multi_wait(CURLM *multi_handle,
                                      struct curl_waitfd extra_fds[],
                                      unsigned int extra_nfds,
                                      int timeout_ms);
2012-09-01 23:10:53 +02:00
Daniel Stenberg
14afbf361a add_next_timeout: minor restructure of code
By reading the ->head pointer and using that instead of the ->size
number to figure out if there's a list remaining we avoid the (false
positive) clang-analyzer warning that we might dereference of a null
pointer.
2012-08-08 14:50:32 +02:00
Daniel Stenberg
9d11716933 multi_runsingle: added precaution against easy_conn NULL pointer
In many states the easy_conn pointer is referenced and just assumed to
be working. This is an added extra check since analyzing indicates
there's a risk we can end up in these states with a NULL pointer there.
2012-07-15 20:33:11 +02:00