Sorry for this being such a huge patch, but I believe it is necessary for
quite a few reasons which I will attempt to explain herein. I've been
mulling this over for a while, but wasn't super happy with making the
download interface more complex. Instead, if we carefully order things in
the internal download code, we can actually make the interface simpler.
1. FS#15657 - This involves `name.db.tar.gz.part` files being left around the
filesystem, and then causing all sorts of issues when someone attempts to
rerun the operation they canceled. We need to ensure that if we resume a
download, we are resuming it on exactly the same file; if we cannot be
almost postive of that then we need to start over.
2. http://www.mail-archive.com/pacman-dev@archlinux.org/msg03536.html - Here
we have a lighttpd bug to ruin the day. If we send both a Range: header and
If-Modified-Since: header across the wire in a GET request, lighttpd doesn't
do what we want in several cases. If the file hadn't been modified, it
returns a '304 Not Modified' instead of a '206 Partial Content'. We need to
do a stat (e.g. HEAD in HTTP terms) operation here, and the proceed
accordingly based off the values we get back from it.
3. The mtime stuff was rather ugly, and relied on the called function to
write back to a passed in reference, which isn't the greatest. Instead, use
the power of the filesystem to contain this info. Every file downloaded
internally is now carefully timestamped with the remote file time. This
should allow the resume logic to work. In order to guarantee this, we need
to implement a signal handler that catches interrupts, notifies the running
code, and causes it to set the mtimes on the file. It then rethrows the
signal so the pacman signal handler (or any frontend) works as expected.
4. We did a lot of funky stuff in trying to track the DB last modified time.
It is a lot easier to just keep the downloaded DB file around and track the
time on that rather than in a funky dot file. It also kills a lot of code.
5. For GPG verification of the databases down the road, we are going to need
the DB file around for at least a short bit of time anyway, so this gets us
closer to that.
Signed-off-by: Dan McGee <dan@archlinux.org>
[Xav: fixed printf with off_t]
Signed-off-by: Xavier Chantry <shiningxc@gmail.com>
* It makes the code clearer to read/understand
* Cppcheck tool doesn't show this anymore: [./util.c:215]: (error) Resource leak: fd
[Dan: don't change the coding style]
Signed-off-by: Dan McGee <dan@archlinux.org>
Thanks to Laszlo Papp <djszapi@archlinux.us> for the following catch:
opendir(path)) == (DIR *)-1;
is maybe the result of misunderstanding the manpage. If an opendir() call
isn't successful it returns NULL rather than '(DIR *)-1'.
Noticed-by: Laszlo Papp <djszapi@archlinux.us>
Signed-off-by: Dan McGee <dan@archlinux.org>
alpm_pkg_get_name() gives us little benefit in backend code besides a NULL
check on the package passed in; we could do that ourself if necessary. By
changing to direct references in the cases where we are sure we have a valid
package, we save a function call each time we need a package name. This
function can't be inlined because it is externally accessible.
This cuts the calls to get_name() from 1.3 million times in a
pacman -Qu operation to around 2400.
Signed-off-by: Dan McGee <dan@archlinux.org>
This fixes the following valgrind warning :
==26831== Syscall param rt_sigaction(act->sa_flags) points to uninitialised
byte(s)
==26831== at 0x4282547: __libc_sigaction (in /lib/libc-2.10.1.so)
==26831== by 0x403C693: download_internal (dload.c:152)
==26831== by 0x403D0E4: _alpm_download_single_file (dload.c:311)
==26831== by 0x4033B72: alpm_db_update (be_files.c:319)
==26831== by 0x805205E: pacman_sync (sync.c:257)
==26831== by 0x804EE54: main (pacman.c:1120)
==26831== Address 0xbec6cc04 is on thread 1's stack
==26831==
==26831== Syscall param rt_sigaction(act->sa_restorer) points to
uninitialised byte(s)
==26831== at 0x4282547: __libc_sigaction (in /lib/libc-2.10.1.so)
==26831== by 0x403C693: download_internal (dload.c:152)
==26831== by 0x403D0E4: _alpm_download_single_file (dload.c:311)
==26831== by 0x4033B72: alpm_db_update (be_files.c:319)
==26831== by 0x805205E: pacman_sync (sync.c:257)
==26831== by 0x804EE54: main (pacman.c:1120)
==26831== Address 0xbec6cc08 is on thread 1's stack
==26831==
Signed-off-by: Xavier Chantry <shiningxc@gmail.com>
Signed-off-by: Dan McGee <dan@archlinux.org>
Pacman's fgets function in the API used hardcoded numbers to identify the size.
This is not good practice, so replace them with sizeof handling.
Signed-off-by: Laszlo Papp <djszapi@archlinux.us>
Signed-off-by: Dan McGee <dan@archlinux.org>
This offers a cleaner way to deal with constant in enum and allow easy
maintainance
Signed-off-by: solsTiCe d'Hiver <solstice.dhiver@gmail.com>
Signed-off-by: Dan McGee <dan@archlinux.org>
If the -d switch was invoked with -S (or -U), the removes list was simply
lost, because trans->remove was computed in an
"if(!(trans->flags & PM_TRANS_FLAG_NODEPS))" block.
I've added a new pactest file, sync045.py (derived from sync043.py) to test
this.
Additionally, I did some other minor cleanups in sync_prepare:
* preferred list is not needed anymore
* I removed a needless alpm_list_remove_dupes line (the target list should
not contain dupes at all)
* I moved alpm_list_free(remove); to cleanup part to eliminate a possible
memleak
Signed-off-by: Nagy Gabor <ngaba@bibl.u-szeged.hu>
Signed-off-by: Dan McGee <dan@archlinux.org>
This is more efficient than alpm_list_diff since it assumes the two lists
are sorted. And also we get the two sides of the diff.
Even sorting should more efficient than the current list_diff. Sorting the
two lists should be O(n*log(n)+m*log(m)) while the current list_diff is
O(n*m). So I also reimplemented list_diff using list_diff_sorted.
Signed-off-by: Xavier Chantry <shiningxc@gmail.com>
Signed-off-by: Dan McGee <dan@archlinux.org>
We don't need to count the number of packages left once per file when
removing; we only need to do it once per package. Also move a variable into
the correct scope.
Signed-off-by: Dan McGee <dan@archlinux.org>
After our recent screwup with size_t and ssize_t in the download code, I
found the `-Wsign-conversion` flag to GCC to see if we were doing anything
else boneheaded. I didn't find anything quite as bad, but we did have some
goofups- most of our public unsigned methods would return -1 on error, which
is a bit odd in an unsigned context.
Signed-off-by: Dan McGee <dan@archlinux.org>
A NULL list element triggered an infinite loop. Not cool :)
Signed-off-by: Xavier Chantry <shiningxc@gmail.com>
Signed-off-by: Dan McGee <dan@archlinux.org>
fetchIO_read returns -1 in case of error, and the return type is
ssize_t, not size_t ! So we converted -1 to an unsigned, which led to
huge file write.
The rest is just changing the error return a bit.
Signed-off-by: Xavier Chantry <shiningxc@gmail.com>
Signed-off-by: Dan McGee <dan@archlinux.org>
Get them a bit more standardized across the board, as they were quite a
mess. Also note the two new translations we received for 3.3.1.
Signed-off-by: Dan McGee <dan@archlinux.org>
This is for 3.3.0, not for 3.3.1. But since there are only like 10 messages
missing, it seems worth including now.
Signed-off-by: Christian Larsson <congacx@gmail.com>
Signed-off-by: Laszlo Papp <djszapi2@archlinux.us>
[Dan: fix some busted translation strings]
Signed-off-by: Dan McGee <dan@archlinux.org>
Commit 34e1413d75 attempted to implement lazy loading of package databases.
Although it took care of my main complaint (creating the database directory
if it didn't exist), it didn't allow sync repos to be registered before
alpm_option_set_dbpath() had been called.
With this patch, we no longer compute the individual repository DB paths
until necessary, allowing full lazy loading to work as intended, and
allowing us to drop the extra setlibpath() calls from the frontend. This
allows the changes introduced in a2cd48960 (but later reverted) to be added
back in again.
Signed-off-by: Dan McGee <dan@archlinux.org>
It was probably a bad idea to modify the target directly in case of
repo/pkg syntax.
Duplicating it also allows us to keep the original target string, which
is more informative when printing errors.
Also remove a duplicated error message from libalpm, and improve the
message already returned to the frontend.
$ pacman -S foo/bar
before
error: repository 'foo' not found
error: 'bar': no such repository
after
error: 'foo/bar': could not find repository for target
Signed-off-by: Xavier Chantry <shiningxc@gmail.com>
Signed-off-by: Dan McGee <dan@archlinux.org>
Add more untranslated strings, improve consistency, etc.
Signed-off-by: Xavier Chantry <shiningxc@gmail.com>
Signed-off-by: Dan McGee <dan@archlinux.org>
After commit 774c252 the --debug output shows 5-6 "syntax error..." lines
for each package. After this patch pacman recognizes makepkgopt as a valid
key, but doesn't do anything.
Signed-off-by: Nagy Gabor <ngaba@bibl.u-szeged.hu>
Signed-off-by: Dan McGee <dan@archlinux.org>
-int alpm_trans_sysupgrade(int enable_downgrade);
-int alpm_trans_sync(char *target);
-int alpm_trans_add(char *target);
-int alpm_trans_remove(char *target);
+int alpm_sync_sysupgrade(int enable_downgrade);
+int alpm_sync_target(char *target);
+int alpm_sync_dbtarget(char *db, char *target);
+int alpm_add_target(char *target);
+int alpm_remove_target(char *target);
* functions renaming
* add new sync_dbtarget which allows to specify the db
* repo/ syntax handling is moved to frontend
( should implement FS#15141)
* group handling is moved to backend
( see http://www.archlinux.org/pipermail/pacman-dev/2009-June/008847.html )
This basically started with this change :
/* Transaction */
struct __pmtrans_t {
- pmtranstype_t type;
pmtransflag_t flags;
pmtransstate_t state;
- alpm_list_t *packages; /* list of (pmpkg_t *) */
+ alpm_list_t *add; /* list of (pmpkg_t *) */
+ alpm_list_t *remove; /* list of (pmpkg_t *) */
And then I have to modify all the code accordingly.
This patch utilizes the power of sync.c to fix FS#3492 and FS#5798.
Now an upgrade transaction is just a sync transaction internally (in alpm),
so all sync features are available with -U as well:
* conflict resolving
* sync dependencies from sync repos
* remove unresolvable targets
See http://www.archlinux.org/pipermail/pacman-dev/2009-June/008725.html
for the concept.
We use "mixed" target list, where PKG_FROM_FILE origin indicates local
package file, PKG_FROM_CACHE indicates sync package. The front-end can add
only one type of packages (depending on transaction type) atm, but if alpm
resolves dependencies for -U, we may get a real mixed trans->packages list.
_alpm_pkg_free_trans() was modified so that it can handle both target types
_alpm_add_prepare() was removed, we use _alpm_sync_prepare() instead
_alpm_add_commit() was renamed to _alpm_upgrade_targets()
sync.c (and deps.c) was modified slightly to handle mixed target lists,
the modifications are straightforward. There is one notable change here: We
don't create new upgrade trans in sync.c, we replace the pkgcache entries
with the loaded package files in the target list (this is a bit hackish) and
call _alpm_upgrade_targets(). This implies a TODO (pkg->origin_data.db is
not accessible anymore), but it doesn't hurt anything with pacman front-end,
so it will be fixed later (otherwise this patch would be huge).
I updated the documentation of -U and I added a new pactest, upgrade090.py,
to test the syncdeps feature of -U.
Signed-off-by: Nagy Gabor <ngaba@bibl.u-szeged.hu>
Signed-off-by: Dan McGee <dan@archlinux.org>
Sometimes "foo conflicts with bar" information is not enough, see this
thread: http://bbs.archlinux.org/viewtopic.php?id=77647. That's why I added
a new reason field to our pmconflict_t struct that stores the packager-
defined conflict that induced the fact that package1 conflicts with
package2.
I modified the front-end (in callback.c, sync.c, upgrade.c) to print this
new information as well.
Signed-off-by: Nagy Gabor <ngaba@bibl.u-szeged.hu>
This function is unused since commit
358cc5804a.
Signed-off-by: Xavier Chantry <shiningxc@gmail.com>
[Dan: also kill from util.h]
Signed-off-by: Dan McGee <dan@archlinux.org>
- fix one memleak if get_filename failed
- cleanup according to Joerg's feedback:
"url_for_string: If fetchParseURL returned successful, you should always
have a scheme set. The logic for anonftp should only be needed for very
broken server -- do you know of any such?
download_internal:
Specifying 'p' is now a nop -- it is tried by default first with
fall-back to active FTP."
Signed-off-by: Xavier Chantry <shiningxc@gmail.com>
[Dan: remove from pacman.conf and pacman.conf.5]
Signed-off-by: Dan McGee <dan@archlinux.org>
libfetch supports checking mtime so we do not need to do it manually.
when the databases are already up-to-date, initiating a connection with
fetchXGet and closing it right after with fetchIO_close took a very long
time (up to 10min!) on some network.
Signed-off-by: Xavier Chantry <shiningxc@gmail.com>
Signed-off-by: Dan McGee <dan@archlinux.org>
(cherry picked from commit d7675e393f)
We had 10000 as our timeout value, assuming it was expressed in ms. This is
false after looking at the current code, so reset it back to 10 seconds.
Addresses FS#15369.
Signed-off-by: Dan McGee <dan@archlinux.org>
I assume the loop was never iterated more than once, because the write
location was not updated at each loop iteration (buffer instead of buffer +
nwritten), yet we never had reports of corrupted download.
Signed-off-by: Xavier Chantry <shiningxc@gmail.com>
Signed-off-by: Dan McGee <dan@archlinux.org>
libfetch supports checking mtime so we do not need to do it manually.
when the databases are already up-to-date, initiating a connection with
fetchXGet and closing it right after with fetchIO_close took a very long
time (up to 10min!) on some network.
Signed-off-by: Xavier Chantry <shiningxc@gmail.com>
Signed-off-by: Dan McGee <dan@archlinux.org>
If /sbin is not in the PATH and sudo is used, ldconfig cannot be found. So
use /sbin/ldconfig instead. The code checked for the existence of
/sbin/ldconfig anyway..
Signed-off-by: Marc - A. Dahlhaus <mad@wol.de>
Signed-off-by: Xavier Chantry <shiningxc@gmail.com>
Signed-off-by: Dan McGee <dan@archlinux.org>
After commit 30c4d53ce5, get_destfile and
get_tempfile are only used for internal download, so move these two
functions inside the ifdef
Signed-off-by: Xavier Chantry <shiningxc@gmail.com>
Signed-off-by: Dan McGee <dan@archlinux.org>
'make distcheck' had issues with this one and reformatted it. In addition,
it found a fuzzy message which is now fixed due to an inadvertent msgid
edit.
Signed-off-by: Dan McGee <dan@archlinux.org>
This allows a frontend to define its own download algorithm so that the
libfetch dependency can be omitted without using an external process.
The callback will be used when if it is defined, otherwise the old
behavior applies.
Signed-off-by: Sebastian Nowicki <sebnow@gmail.com>
[Dan: minor cleanups]
Signed-off-by: Dan McGee <dan@archlinux.org>
If the user switches from unstable repo to a stable one, it is quite hard to
sync its system with the new repo (the user will see many "Local is newer
than stable" messages, nothing more). That's why I introduced -Suu, which
treats a sync package like an upgrade, iff the package version doesn't match
with the local one's.
I added a new pactest (sync104.py) to test this, and I updated the
documentation of -Su.
Signed-off-by: Nagy Gabor <ngaba@bibl.u-szeged.hu>
[Dan: slight doc reword]
Signed-off-by: Dan McGee <dan@archlinux.org>
This happens for example if you install a new package, and one of its
backup config file is already on the file system.
If the local file was different, it was saved to .pacorig which is fine.
However if the local file and pkg file were the same, the pkg file
(temporarily extracted as .paccheck) was left on the system.
Signed-off-by: Xavier Chantry <shiningxc@gmail.com>
Signed-off-by: Dan McGee <dan@archlinux.org>
This fixes FS#15546
Also fix the interface of unlink_file which was really stupid..
(alpm_list_t used with only one element)
Signed-off-by: Xavier Chantry <shiningxc@gmail.com>
Signed-off-by: Dan McGee <dan@archlinux.org>
A package can now replace symdir->dir by dir without fileconflicts.
Signed-off-by: Xavier Chantry <shiningxc@gmail.com>
Signed-off-by: Dan McGee <dan@archlinux.org>
