Commit Graph

514 Commits

Author SHA1 Message Date
Daniel Stenberg 8a49f91d32
multi: make the closure handle "inherit" CURLOPT_NOSIGNAL
Otherwise, closing that handle can still cause surprises!

Reported-by: Martin Ankerl
Fixes #3138
Closes #3147
2018-10-19 11:03:17 +02:00
Michael Kaufmann 6afe70a00b Curl_follow: Always free the passed new URL
Closes #3124
2018-10-13 13:18:51 +02:00
Daniel Stenberg 2dfc0dd6b5
multi: fix memory leak in content encoding related error path
... a missing multi_done() call.

Credit to OSS-Fuzz
Bug: https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=10728
Closes #3063
2018-09-29 15:03:57 +02:00
Daniel Stenberg 454fa3fd7b
multi: fix location URL memleak in error path
Follow-up to #3044 - fix a leak OSS-Fuzz detected
Closes #3057
2018-09-28 17:10:14 +02:00
Daniel Stenberg 46e164069d
url: use the URL API internally as well
... to make it a truly unified URL parser.

Closes #3017
2018-09-22 11:58:10 +02:00
Daniel Stenberg ec5d0991cc
curl_multi_wait: call getsock before figuring out timeout
.... since getsock may update the expiry timer.

Fixes #2996
Closes #3000
2018-09-18 15:39:28 +02:00
Daniel Stenberg abff183387
setopt: add CURLOPT_DOH_URL
Closes #2668
2018-09-06 09:17:17 +02:00
Daniel Stenberg d7c4213bd0
multiplex: enable by default
Starting 7.62.0, multiplexing is enabled by default in multi handles.
2018-09-05 14:35:57 +02:00
Daniel Stenberg f7208df7d9
pipelining: deprecated
Transparently. The related curl_multi_setopt() options all still returns
OK when pipelining is selected.

To re-enable the support, the single line change in lib/multi.c needs to
be reverted.

See docs/DEPRECATE.md

Closes #2705
2018-09-05 11:34:03 +02:00
Daniel Stenberg e6e9b006f7
upload: allocate upload buffer on-demand
Saves 16KB on the easy handle for operations that don't need that
buffer.

Part 1 of #2888
2018-08-18 15:48:08 +02:00
Daniel Stenberg 09da53998e
urldata: remove unused pipe_broke struct field
This struct field is never set TRUE in any existing code path. This
change removes the field completely.

Closes #2871
2018-08-16 09:33:12 +02:00
Daniel Jelinski d6cf93011d
retry: return error if rewind was necessary but didn't happen
Fixes #2801
Closes #2812
2018-07-31 01:26:08 +02:00
Daniel Stenberg acefdd0cd1
multi: always do the COMPLETED procedure/state
It was previously erroneously skipped in some situations.

libtest/libntlmconnect.c wrongly depended on wrong behavior (that it
would get a zero timeout) when no handles are "running" in a multi
handle. That behavior is no longer present with this fix. Now libcurl
will always return a -1 timeout when all handles are completed.

Closes #2733
2018-07-11 23:41:24 +02:00
Daniel Stenberg 1b76c38904
conn: remove the boolean 'inuse' field
... as the usage needs to be counted.
2018-07-11 23:41:24 +02:00
Javier Blazquez 4c901638b4
multi: fix crash due to dangling entry in connect-pending list
Fixes #2677
Closes #2679
2018-06-23 22:31:52 +02:00
Daniel Stenberg ac9a179fe9
multi: fix memory leak when stopped during name resolve
When the application just started the transfer and then stops it while
the name resolve in the background thread hasn't completed, we need to
wait for the resolve to complete and then cleanup data accordingly.

Enabled test 1553 again and added test 1590 to also check when the host
name resolves successfully.

Detected by OSS-fuzz.
Closes #1968
2018-06-16 12:01:27 +02:00
Marian Klymov c45360d463
cppcheck: fix warnings
- Get rid of variable that was generating false positive warning
(unitialized)

- Fix issues in tests

- Reduce scope of several variables all over

etc

Closes #2631
2018-06-11 11:14:48 +02:00
Viktor Szakats 4bd91bc474 spelling fixes
Detected using the `codespell` tool (version 1.13.0).

Also secure and fix an URL.
2018-06-03 12:14:45 +00:00
Daniel Stenberg 923ce4a590
multi: remove a DEBUGF()
... it might call infof() with a NULL first argument that isn't harmful
but makes it not do anything. The infof() line is not very useful
anymore, it has served it purpose. Good riddance!

Fixes #2627
2018-06-01 12:55:41 +02:00
Rikard Falkeborn 8ea5d41fe0
strictness: correct {infof, failf} format specifiers
Closes #2623
2018-05-31 11:34:32 +02:00
Patrick Monnerat 8541d02c96 psl: use latest psl and refresh it periodically
The latest psl is cached in the multi or share handle. It is refreshed
before use after 72 hours.
New share lock CURL_LOCK_DATA_PSL controls the psl cache sharing.
If the latest psl is not available, the builtin psl is used.

Reported-by: Yaakov Selkowitz
Fixes #2553
Closes #2601
2018-05-28 20:37:14 +02:00
Daniel Gustafsson 2f13e3d23d
checksrc: force indentation of lines after an else
This extends the INDENTATION case to also handle 'else' statements
and require proper indentation on the following line. Also fixes the
offending cases found in the codebase.

Closes #2532
2018-04-27 00:51:35 +02:00
Daniel Stenberg d122df5972
http2: handle GOAWAY properly
When receiving REFUSED_STREAM, mark the connection for close and retry
streams accordingly on another/fresh connection.

Reported-by: Terry Wu
Fixes #2416
Fixes #1618
Closes #2510
2018-04-20 22:19:23 +02:00
Daniel Stenberg fb4f568b1e
add_handle/easy_perform: clear errorbuffer on start if set
To offer applications a more defined behavior, we clear the buffer as
early as possible.

Assisted-by: Jay Satiro

Fixes #2190
Closes #2377
2018-03-17 12:07:37 +01:00
Daniel Stenberg 7f9ce0851a
multi: improved pending transfers handling => improved performance
When a transfer is requested to get done and it is put in the pending
queue when limited by number of connections, total or per-host, libcurl
would previously very aggressively retry *ALL* pending transfers to get
them transferring. That was very time consuming.

By reducing the aggressiveness in how pending are being retried, we
waste MUCH less time on putting transfers back into pending again.

Some test cases got a factor 30(!) speed improvement with this change.

Reported-by: Cyril B
Fixes #2369
Closes #2383
2018-03-16 23:45:59 +01:00
Daniel Stenberg 2404aa080e
pause: when changing pause state, update socket state
Especially unpausing a transfer might have to move the socket back to the
"currently used sockets" hash to get monitored. Otherwise it would never get
any more data and get stuck. Easily triggered with pausing using the
multi_socket API.

Reported-by: Philip Prindeville
Bug: https://curl.haxx.se/mail/lib-2018-03/0048.html
Fixes #2393
Closes #2391
2018-03-16 23:41:55 +01:00
Daniel Stenberg f5700ea88b
rate-limit: use three second window to better handle high speeds
Due to very frequent updates of the rate limit "window", it could
attempt to rate limit within the same milliseconds and that then made
the calculations wrong, leading to it not behaving correctly on very
fast transfers.

This new logic updates the rate limit "window" to be no shorter than the
last three seconds and only updating the timestamps for this when
switching between the states TOOFAST/PERFORM.

Reported-by: 刘佩东
Fixes #2386
Closes #2388
2018-03-16 11:10:06 +01:00
Daniel Stenberg 8b498a875c
http2: mark the connection for close on GOAWAY
... don't consider it an error!

Assisted-by: Jay Satiro
Reported-by: Łukasz Domeradzki
Fixes #2365
Closes #2375
2018-03-12 08:07:42 +01:00
Björn Stenberg b46cfbc068
TODO fixed: Detect when called from within callbacks
Closes #2302
2018-02-15 09:36:03 +01:00
Kartik Mahajan a5e6d6ebcb
http2: don't close connection when single transfer is stopped
Fixes #2237
Closes #2249
2018-01-18 22:28:12 +01:00
Daniel Stenberg e959f16c5f
multi_done: prune DNS cache
Prune the DNS cache immediately after the dns entry is unlocked in
multi_done. Timed out entries will then get discarded in a more orderly
fashion.

Test506 is updated

Reported-by: Oleg Pudeyev

Fixes #2169
Closes #2170
2017-12-10 14:51:47 +01:00
Daniel Stenberg 07cb27c98e
conncache: fix several lock issues
If the lock is released before the dealings with the bundle is over, it may
have changed by another thread in the mean time.

Fixes #2132
Fixes #2151
Closes #2139
2017-12-05 23:21:02 +01:00
Daniel Stenberg 67c55a26d5
share: add support for sharing the connection cache 2017-11-09 11:07:44 +01:00
Daniel Stenberg 0d85eed3df
Curl_timeleft: change return type to timediff_t
returning 'time_t' is problematic when that type is unsigned and we
return values less than zero to signal "already expired", used in
several places in the code.

Closes #2021
2017-10-28 10:40:51 +02:00
Max Dymond 7b11c5dbe6
wildcards: don't use with non-supported protocols
Fixes timeouts in the fuzzing tests for non-FTP protocols.

Closes #2016
2017-10-26 13:34:45 +02:00
Max Dymond 3340b456a5 multi: allow table handle sizes to be overridden
Allow users to specify their own hash define for
CURL_CONNECTION_HASH_SIZE so that both values can be overridden.

Closes #1982
2017-10-25 18:50:02 +02:00
Daniel Stenberg 5d543fe906 time: rename Curl_tvnow to Curl_now
... since the 'tv' stood for timeval and this function does not return a
timeval struct anymore.

Also, cleaned up the Curl_timediff*() functions to avoid typecasts and
clean up the descriptive comments.

Closes #2011
2017-10-25 18:48:05 +02:00
Daniel Stenberg b9d25f9a6b timediff: return timediff_t from the time diff functions
... to cater for systems with unsigned time_t variables.

- Renamed the functions to curlx_timediff and Curl_timediff_us.

- Added overflow protection for both of them in either direction for
  both 32 bit and 64 bit time_ts

- Reprefixed the curlx_time functions to use Curl_*

Reported-by: Peter Piekarski
Fixes #2004
Closes #2005
2017-10-25 09:54:37 +02:00
Daniel Stenberg b2df2d47e5
Revert "multi_done: wait for name resolve to finish if still ongoing"
This reverts commit f3e03f6c0a.

Caused memory leaks in the fuzzer, needs to be done differently.

Disable test 1553 for now too, as it causes memory leaks without this
commit!
2017-10-08 00:55:10 +02:00
Daniel Stenberg 1e552535e1
remove_handle: call multi_done() first, then clear dns cache pointer
Closes #1960
2017-10-07 23:54:33 +02:00
Daniel Stenberg f3e03f6c0a
multi_done: wait for name resolve to finish if still ongoing
... as we must clean up memory.
2017-10-07 17:54:41 +02:00
Daniel Stenberg 7f1140c8bf
multi_cleanup: call DONE on handles that never got that
... fixes a memory leak with at least IMAP when remove_handle is never
called and the transfer is abruptly just abandoned early.

Test 1552 added to verify

Detected by OSS-fuzz
Assisted-by: Max Dymond
Closes #1954
2017-10-06 16:48:39 +02:00
Daniel Stenberg e5743f08e7
code style: use spaces around pluses 2017-09-11 09:29:50 +02:00
Daniel Stenberg 6b84438d9a
code style: use spaces around equals signs 2017-09-11 09:29:50 +02:00
Brad Spencer 164a09368d multi: fix request timer management
There are some bugs in how timers are managed for a single easy handle
that causes the wrong "next timeout" value to be reported to the
application when a new minimum needs to be recomputed and that new
minimum should be an existing timer that isn't currently set for the
easy handle.  When the application drives a set of easy handles via the
`curl_multi_socket_action()` API (for example), it gets told to wait the
wrong amount of time before the next call, which causes requests to
linger for a long time (or, it is my guess, possibly forever).

Bug: https://curl.haxx.se/mail/lib-2017-07/0033.html
2017-08-01 13:39:38 +02:00
Daniel Stenberg 4dee50b9c8 timeval: struct curltime is a struct timeval replacement
... to make all libcurl internals able to use the same data types for
the struct members. The timeval struct differs subtly on several
platforms so it makes it cumbersome to use everywhere.

Ref: #1652
Closes #1693
2017-07-28 15:51:25 +02:00
Daniel Stenberg 02c7a2ccab multi: mention integer overflow risk if using > 500 million sockets
Reported-by: ovidiu-benea@users.noreply.github.com

Closes #1675
Closes #1683
2017-07-27 01:13:47 +02:00
Daniel Stenberg 5113ad0424 http-proxy: do the HTTP CONNECT process entirely non-blocking
Mentioned as a problem since 2007 (8f87c15bda) and of course it
existed even before that.

Closes #1547
2017-06-14 23:43:52 +02:00
Daniel Stenberg 7fffe97b78 expire: remove Curl_expire_latest()
With the introduction of expire IDs and the fact that existing timers
can be removed now and thus never expire, the concept with adding a
"latest" timer is not working anymore as it risks to not expire at all.

So, to be certain the timers actually are in line and will expire, the
plain Curl_expire() needs to be used. The _latest() function was added
as a sort of shortcut in the past that's quite simply not necessary
anymore.

Follow-up to 31b39c40cf

Reported-by: Paul Harris

Closes #1555
2017-06-08 10:50:47 +02:00
Daniel Stenberg bba59073c5 redirect: store the "would redirect to" URL when max redirs is reached
Test 1261 added to verify.

Reported-by: Lloyd Fournier

Fixes #1489
Closes #1497
2017-05-23 23:27:58 +02:00