1
0
mirror of https://github.com/moparisthebest/curl synced 2024-12-25 09:38:54 -05:00
Commit Graph

496 Commits

Author SHA1 Message Date
Daniel Stenberg
923ce4a590
multi: remove a DEBUGF()
... it might call infof() with a NULL first argument that isn't harmful
but makes it not do anything. The infof() line is not very useful
anymore, it has served it purpose. Good riddance!

Fixes #2627
2018-06-01 12:55:41 +02:00
Rikard Falkeborn
8ea5d41fe0
strictness: correct {infof, failf} format specifiers
Closes #2623
2018-05-31 11:34:32 +02:00
Patrick Monnerat
8541d02c96 psl: use latest psl and refresh it periodically
The latest psl is cached in the multi or share handle. It is refreshed
before use after 72 hours.
New share lock CURL_LOCK_DATA_PSL controls the psl cache sharing.
If the latest psl is not available, the builtin psl is used.

Reported-by: Yaakov Selkowitz
Fixes #2553
Closes #2601
2018-05-28 20:37:14 +02:00
Daniel Gustafsson
2f13e3d23d
checksrc: force indentation of lines after an else
This extends the INDENTATION case to also handle 'else' statements
and require proper indentation on the following line. Also fixes the
offending cases found in the codebase.

Closes #2532
2018-04-27 00:51:35 +02:00
Daniel Stenberg
d122df5972
http2: handle GOAWAY properly
When receiving REFUSED_STREAM, mark the connection for close and retry
streams accordingly on another/fresh connection.

Reported-by: Terry Wu
Fixes #2416
Fixes #1618
Closes #2510
2018-04-20 22:19:23 +02:00
Daniel Stenberg
fb4f568b1e
add_handle/easy_perform: clear errorbuffer on start if set
To offer applications a more defined behavior, we clear the buffer as
early as possible.

Assisted-by: Jay Satiro

Fixes #2190
Closes #2377
2018-03-17 12:07:37 +01:00
Daniel Stenberg
7f9ce0851a
multi: improved pending transfers handling => improved performance
When a transfer is requested to get done and it is put in the pending
queue when limited by number of connections, total or per-host, libcurl
would previously very aggressively retry *ALL* pending transfers to get
them transferring. That was very time consuming.

By reducing the aggressiveness in how pending are being retried, we
waste MUCH less time on putting transfers back into pending again.

Some test cases got a factor 30(!) speed improvement with this change.

Reported-by: Cyril B
Fixes #2369
Closes #2383
2018-03-16 23:45:59 +01:00
Daniel Stenberg
2404aa080e
pause: when changing pause state, update socket state
Especially unpausing a transfer might have to move the socket back to the
"currently used sockets" hash to get monitored. Otherwise it would never get
any more data and get stuck. Easily triggered with pausing using the
multi_socket API.

Reported-by: Philip Prindeville
Bug: https://curl.haxx.se/mail/lib-2018-03/0048.html
Fixes #2393
Closes #2391
2018-03-16 23:41:55 +01:00
Daniel Stenberg
f5700ea88b
rate-limit: use three second window to better handle high speeds
Due to very frequent updates of the rate limit "window", it could
attempt to rate limit within the same milliseconds and that then made
the calculations wrong, leading to it not behaving correctly on very
fast transfers.

This new logic updates the rate limit "window" to be no shorter than the
last three seconds and only updating the timestamps for this when
switching between the states TOOFAST/PERFORM.

Reported-by: 刘佩东
Fixes #2386
Closes #2388
2018-03-16 11:10:06 +01:00
Daniel Stenberg
8b498a875c
http2: mark the connection for close on GOAWAY
... don't consider it an error!

Assisted-by: Jay Satiro
Reported-by: Łukasz Domeradzki
Fixes #2365
Closes #2375
2018-03-12 08:07:42 +01:00
Björn Stenberg
b46cfbc068
TODO fixed: Detect when called from within callbacks
Closes #2302
2018-02-15 09:36:03 +01:00
Kartik Mahajan
a5e6d6ebcb
http2: don't close connection when single transfer is stopped
Fixes #2237
Closes #2249
2018-01-18 22:28:12 +01:00
Daniel Stenberg
e959f16c5f
multi_done: prune DNS cache
Prune the DNS cache immediately after the dns entry is unlocked in
multi_done. Timed out entries will then get discarded in a more orderly
fashion.

Test506 is updated

Reported-by: Oleg Pudeyev

Fixes #2169
Closes #2170
2017-12-10 14:51:47 +01:00
Daniel Stenberg
07cb27c98e
conncache: fix several lock issues
If the lock is released before the dealings with the bundle is over, it may
have changed by another thread in the mean time.

Fixes #2132
Fixes #2151
Closes #2139
2017-12-05 23:21:02 +01:00
Daniel Stenberg
67c55a26d5
share: add support for sharing the connection cache 2017-11-09 11:07:44 +01:00
Daniel Stenberg
0d85eed3df
Curl_timeleft: change return type to timediff_t
returning 'time_t' is problematic when that type is unsigned and we
return values less than zero to signal "already expired", used in
several places in the code.

Closes #2021
2017-10-28 10:40:51 +02:00
Max Dymond
7b11c5dbe6
wildcards: don't use with non-supported protocols
Fixes timeouts in the fuzzing tests for non-FTP protocols.

Closes #2016
2017-10-26 13:34:45 +02:00
Max Dymond
3340b456a5 multi: allow table handle sizes to be overridden
Allow users to specify their own hash define for
CURL_CONNECTION_HASH_SIZE so that both values can be overridden.

Closes #1982
2017-10-25 18:50:02 +02:00
Daniel Stenberg
5d543fe906 time: rename Curl_tvnow to Curl_now
... since the 'tv' stood for timeval and this function does not return a
timeval struct anymore.

Also, cleaned up the Curl_timediff*() functions to avoid typecasts and
clean up the descriptive comments.

Closes #2011
2017-10-25 18:48:05 +02:00
Daniel Stenberg
b9d25f9a6b timediff: return timediff_t from the time diff functions
... to cater for systems with unsigned time_t variables.

- Renamed the functions to curlx_timediff and Curl_timediff_us.

- Added overflow protection for both of them in either direction for
  both 32 bit and 64 bit time_ts

- Reprefixed the curlx_time functions to use Curl_*

Reported-by: Peter Piekarski
Fixes #2004
Closes #2005
2017-10-25 09:54:37 +02:00
Daniel Stenberg
b2df2d47e5
Revert "multi_done: wait for name resolve to finish if still ongoing"
This reverts commit f3e03f6c0a.

Caused memory leaks in the fuzzer, needs to be done differently.

Disable test 1553 for now too, as it causes memory leaks without this
commit!
2017-10-08 00:55:10 +02:00
Daniel Stenberg
1e552535e1
remove_handle: call multi_done() first, then clear dns cache pointer
Closes #1960
2017-10-07 23:54:33 +02:00
Daniel Stenberg
f3e03f6c0a
multi_done: wait for name resolve to finish if still ongoing
... as we must clean up memory.
2017-10-07 17:54:41 +02:00
Daniel Stenberg
7f1140c8bf
multi_cleanup: call DONE on handles that never got that
... fixes a memory leak with at least IMAP when remove_handle is never
called and the transfer is abruptly just abandoned early.

Test 1552 added to verify

Detected by OSS-fuzz
Assisted-by: Max Dymond
Closes #1954
2017-10-06 16:48:39 +02:00
Daniel Stenberg
e5743f08e7
code style: use spaces around pluses 2017-09-11 09:29:50 +02:00
Daniel Stenberg
6b84438d9a
code style: use spaces around equals signs 2017-09-11 09:29:50 +02:00
Brad Spencer
164a09368d multi: fix request timer management
There are some bugs in how timers are managed for a single easy handle
that causes the wrong "next timeout" value to be reported to the
application when a new minimum needs to be recomputed and that new
minimum should be an existing timer that isn't currently set for the
easy handle.  When the application drives a set of easy handles via the
`curl_multi_socket_action()` API (for example), it gets told to wait the
wrong amount of time before the next call, which causes requests to
linger for a long time (or, it is my guess, possibly forever).

Bug: https://curl.haxx.se/mail/lib-2017-07/0033.html
2017-08-01 13:39:38 +02:00
Daniel Stenberg
4dee50b9c8 timeval: struct curltime is a struct timeval replacement
... to make all libcurl internals able to use the same data types for
the struct members. The timeval struct differs subtly on several
platforms so it makes it cumbersome to use everywhere.

Ref: #1652
Closes #1693
2017-07-28 15:51:25 +02:00
Daniel Stenberg
02c7a2ccab multi: mention integer overflow risk if using > 500 million sockets
Reported-by: ovidiu-benea@users.noreply.github.com

Closes #1675
Closes #1683
2017-07-27 01:13:47 +02:00
Daniel Stenberg
5113ad0424 http-proxy: do the HTTP CONNECT process entirely non-blocking
Mentioned as a problem since 2007 (8f87c15bda) and of course it
existed even before that.

Closes #1547
2017-06-14 23:43:52 +02:00
Daniel Stenberg
7fffe97b78 expire: remove Curl_expire_latest()
With the introduction of expire IDs and the fact that existing timers
can be removed now and thus never expire, the concept with adding a
"latest" timer is not working anymore as it risks to not expire at all.

So, to be certain the timers actually are in line and will expire, the
plain Curl_expire() needs to be used. The _latest() function was added
as a sort of shortcut in the past that's quite simply not necessary
anymore.

Follow-up to 31b39c40cf

Reported-by: Paul Harris

Closes #1555
2017-06-08 10:50:47 +02:00
Daniel Stenberg
bba59073c5 redirect: store the "would redirect to" URL when max redirs is reached
Test 1261 added to verify.

Reported-by: Lloyd Fournier

Fixes #1489
Closes #1497
2017-05-23 23:27:58 +02:00
Daniel Stenberg
d6c74ff663 multi: remove leftover debug infof() calls from e9fd794a6 2017-05-12 17:19:30 +02:00
Daniel Stenberg
31b39c40cf multi: use a fixed array of timers instead of malloc
... since the total amount is low this is faster, easier and reduces
memory overhead.

Also, Curl_expire_done() can now mark an expire timeout as done so that
it never times out.

Closes #1472
2017-05-10 12:55:36 +02:00
Daniel Stenberg
e9fd794a61 multi: assign IDs to all timers and make each timer singleton
A) reduces the timeout lists drastically

 B) prevents a lot of superfluous loops for timers that expires "in vain"
    when it has actually already been extended to fire later on
2017-05-10 11:02:47 +02:00
Alan Jenkins
be299a4dba multi: clarify condition in curl_multi_wait
`if(nfds || extra_nfds) {` is followed by `malloc(nfds * ...)`.

If `extra_fs` could be non-zero when `nfds` was zero, then we have
`malloc(0)` which is allowed to return `NULL`. But, malloc returning
NULL can be confusing. In this code, the next line would treat the NULL
as an allocation failure.

It turns out, if `nfds` is zero then `extra_nfds` must also be zero.
The final value of `nfds` includes `extra_nfds`.  So the test for
`extra_nfds` is redundant.  It can only confuse the reader.

Closes #1439
2017-04-22 22:35:46 +02:00
Daniel Stenberg
cbae73e1dd llist: no longer uses malloc
The 'list element' struct now has to be within the data that is being
added to the list. Removes 16.6% (tiny) mallocs from a simple HTTP
transfer. (96 => 80)

Also removed return codes since the llist functions can't fail now.

Test 1300 updated accordingly.

Closes #1435
2017-04-22 11:25:27 +02:00
Daniel Stenberg
ab6d23278e Curl_expire_latest: ignore already expired timers
If the existing timer is still in there but has expired, the new timer
should be added.

Reported-by: Rainer Canavan
Bug: https://curl.haxx.se/mail/lib-2017-04/0030.html
Closes #1407
2017-04-11 16:53:33 +02:00
Daniel Stenberg
e60fe20fdf llist: replace Curl_llist_alloc with Curl_llist_init
No longer allocate the curl_llist head struct for lists separately.

Removes 17 (15%) tiny allocations in a normal "curl localhost" invoke.

closes #1381
2017-04-04 15:27:45 +02:00
Daniel Stenberg
5f1163517e multi: make curl_multi_wait avoid malloc in the typical case
When only a few additional file descriptors are used, avoid the malloc.

Closes #1377
2017-04-03 23:20:02 +02:00
Daniel Stenberg
452203341d pause: handle mixed types of data when paused
When receiving chunked encoded data with trailers, and the write
callback returns PAUSE, there might be both body and header to store to
resend on unpause. Previously libcurl returned error for that case.

Added test case 1540 to verify.

Reported-by: Stephen Toub
Fixes #1354
Closes #1357
2017-03-28 13:27:49 +02:00
Marcel Raad
068cd1ad4c
multi: fix MinGW-w64 compiler warnings
error: conversion to 'long int' from 'time_t {aka long long int}' may alter
its value [-Werror=conversion]
2017-03-27 22:50:59 +02:00
Daniel Stenberg
6e0f26c8a8 multi: fix streamclose() crash in debug mode
The code would refer to the wrong data pointer. Only debug builds do
this - for verbosity.

Reported-by: zelinchen@users.noreply.github.com
Fixes #1329
2017-03-21 10:09:11 +01:00
Sylvestre Ledru
66de563482 Improve code readbility
... by removing the else branch after a return, break or continue.

Closes #1310
2017-03-13 23:11:45 +01:00
Michael Kaufmann
a5e8fcbc17 speed caps: update the timeouts if the speed is too low/high
Follow-up to 4b86113

Fixes https://github.com/curl/curl/issues/793
Fixes https://github.com/curl/curl/issues/942
2017-02-18 18:09:17 +01:00
Michael Kaufmann
2f8d0df085 proxy: fix hostname resolution and IDN conversion
Properly resolve, convert and log the proxy host names.
Support the "--connect-to" feature for SOCKS proxies and for passive FTP
data transfers.

Follow-up to cb4e2be

Reported-by: Jay Satiro
Fixes https://github.com/curl/curl/issues/1248
2017-02-18 15:04:43 +01:00
Okhin Vasilij
c6da05a5ec HTTPS-proxy: fixed mbedtls and polishing 2016-11-24 23:41:45 +01:00
Alex Rousskov
cb4e2be7c6 proxy: Support HTTPS proxy and SOCKS+HTTP(s)
* HTTPS proxies:

An HTTPS proxy receives all transactions over an SSL/TLS connection.
Once a secure connection with the proxy is established, the user agent
uses the proxy as usual, including sending CONNECT requests to instruct
the proxy to establish a [usually secure] TCP tunnel with an origin
server. HTTPS proxies protect nearly all aspects of user-proxy
communications as opposed to HTTP proxies that receive all requests
(including CONNECT requests) in vulnerable clear text.

With HTTPS proxies, it is possible to have two concurrent _nested_
SSL/TLS sessions: the "outer" one between the user agent and the proxy
and the "inner" one between the user agent and the origin server
(through the proxy). This change adds supports for such nested sessions
as well.

A secure connection with a proxy requires its own set of the usual SSL
options (their actual descriptions differ and need polishing, see TODO):

  --proxy-cacert FILE        CA certificate to verify peer against
  --proxy-capath DIR         CA directory to verify peer against
  --proxy-cert CERT[:PASSWD] Client certificate file and password
  --proxy-cert-type TYPE     Certificate file type (DER/PEM/ENG)
  --proxy-ciphers LIST       SSL ciphers to use
  --proxy-crlfile FILE       Get a CRL list in PEM format from the file
  --proxy-insecure           Allow connections to proxies with bad certs
  --proxy-key KEY            Private key file name
  --proxy-key-type TYPE      Private key file type (DER/PEM/ENG)
  --proxy-pass PASS          Pass phrase for the private key
  --proxy-ssl-allow-beast    Allow security flaw to improve interop
  --proxy-sslv2              Use SSLv2
  --proxy-sslv3              Use SSLv3
  --proxy-tlsv1              Use TLSv1
  --proxy-tlsuser USER       TLS username
  --proxy-tlspassword STRING TLS password
  --proxy-tlsauthtype STRING TLS authentication type (default SRP)

All --proxy-foo options are independent from their --foo counterparts,
except --proxy-crlfile which defaults to --crlfile and --proxy-capath
which defaults to --capath.

Curl now also supports %{proxy_ssl_verify_result} --write-out variable,
similar to the existing %{ssl_verify_result} variable.

Supported backends: OpenSSL, GnuTLS, and NSS.

* A SOCKS proxy + HTTP/HTTPS proxy combination:

If both --socks* and --proxy options are given, Curl first connects to
the SOCKS proxy and then connects (through SOCKS) to the HTTP or HTTPS
proxy.

TODO: Update documentation for the new APIs and --proxy-* options.
Look for "Added in 7.XXX" marks.
2016-11-24 23:41:44 +01:00
Marcel Raad
21aa32d30d lib: fix compiler warnings after de4de4e3c7
Visual C++ now complains about implicitly casting time_t (64-bit) to
long (32-bit). Fix this by changing some variables from long to time_t,
or explicitly casting to long where the public interface would be
affected.

Closes #1131
2016-11-18 10:11:55 +01:00
Daniel Stenberg
6290c3c26b multi: force connections to get closed in close_all_connections
Several independent reports on infinite loops hanging in the
close_all_connections() function when closing a multi handle, can be
fixed by first marking the connection to get closed before calling
Curl_disconnect.

This is more fixing-the-symptom rather than the underlying problem
though.

Bug: https://curl.haxx.se/mail/lib-2016-10/0011.html
Bug: https://curl.haxx.se/mail/lib-2016-10/0059.html

Reported-by: Dan Fandrich, Valentin David, Miloš Ljumović
2016-10-22 16:10:57 +02:00