Sorry for this being such a huge patch, but I believe it is necessary for
quite a few reasons which I will attempt to explain herein. I've been
mulling this over for a while, but wasn't super happy with making the
download interface more complex. Instead, if we carefully order things in
the internal download code, we can actually make the interface simpler.
1. FS#15657 - This involves `name.db.tar.gz.part` files being left around the
filesystem, and then causing all sorts of issues when someone attempts to
rerun the operation they canceled. We need to ensure that if we resume a
download, we are resuming it on exactly the same file; if we cannot be
almost postive of that then we need to start over.
2. http://www.mail-archive.com/pacman-dev@archlinux.org/msg03536.html - Here
we have a lighttpd bug to ruin the day. If we send both a Range: header and
If-Modified-Since: header across the wire in a GET request, lighttpd doesn't
do what we want in several cases. If the file hadn't been modified, it
returns a '304 Not Modified' instead of a '206 Partial Content'. We need to
do a stat (e.g. HEAD in HTTP terms) operation here, and the proceed
accordingly based off the values we get back from it.
3. The mtime stuff was rather ugly, and relied on the called function to
write back to a passed in reference, which isn't the greatest. Instead, use
the power of the filesystem to contain this info. Every file downloaded
internally is now carefully timestamped with the remote file time. This
should allow the resume logic to work. In order to guarantee this, we need
to implement a signal handler that catches interrupts, notifies the running
code, and causes it to set the mtimes on the file. It then rethrows the
signal so the pacman signal handler (or any frontend) works as expected.
4. We did a lot of funky stuff in trying to track the DB last modified time.
It is a lot easier to just keep the downloaded DB file around and track the
time on that rather than in a funky dot file. It also kills a lot of code.
5. For GPG verification of the databases down the road, we are going to need
the DB file around for at least a short bit of time anyway, so this gets us
closer to that.
Signed-off-by: Dan McGee <dan@archlinux.org>
[Xav: fixed printf with off_t]
Signed-off-by: Xavier Chantry <shiningxc@gmail.com>
This offers a cleaner way to deal with constant in enum and allow easy
maintainance
Signed-off-by: solsTiCe d'Hiver <solstice.dhiver@gmail.com>
Signed-off-by: Dan McGee <dan@archlinux.org>
After our recent screwup with size_t and ssize_t in the download code, I
found the `-Wsign-conversion` flag to GCC to see if we were doing anything
else boneheaded. I didn't find anything quite as bad, but we did have some
goofups- most of our public unsigned methods would return -1 on error, which
is a bit odd in an unsigned context.
Signed-off-by: Dan McGee <dan@archlinux.org>
-int alpm_trans_sysupgrade(int enable_downgrade);
-int alpm_trans_sync(char *target);
-int alpm_trans_add(char *target);
-int alpm_trans_remove(char *target);
+int alpm_sync_sysupgrade(int enable_downgrade);
+int alpm_sync_target(char *target);
+int alpm_sync_dbtarget(char *db, char *target);
+int alpm_add_target(char *target);
+int alpm_remove_target(char *target);
* functions renaming
* add new sync_dbtarget which allows to specify the db
* repo/ syntax handling is moved to frontend
( should implement FS#15141)
* group handling is moved to backend
( see http://www.archlinux.org/pipermail/pacman-dev/2009-June/008847.html )
This basically started with this change :
/* Transaction */
struct __pmtrans_t {
- pmtranstype_t type;
pmtransflag_t flags;
pmtransstate_t state;
- alpm_list_t *packages; /* list of (pmpkg_t *) */
+ alpm_list_t *add; /* list of (pmpkg_t *) */
+ alpm_list_t *remove; /* list of (pmpkg_t *) */
And then I have to modify all the code accordingly.
Sometimes "foo conflicts with bar" information is not enough, see this
thread: http://bbs.archlinux.org/viewtopic.php?id=77647. That's why I added
a new reason field to our pmconflict_t struct that stores the packager-
defined conflict that induced the fact that package1 conflicts with
package2.
I modified the front-end (in callback.c, sync.c, upgrade.c) to print this
new information as well.
Signed-off-by: Nagy Gabor <ngaba@bibl.u-szeged.hu>
- fix one memleak if get_filename failed
- cleanup according to Joerg's feedback:
"url_for_string: If fetchParseURL returned successful, you should always
have a scheme set. The logic for anonftp should only be needed for very
broken server -- do you know of any such?
download_internal:
Specifying 'p' is now a nop -- it is tried by default first with
fall-back to active FTP."
Signed-off-by: Xavier Chantry <shiningxc@gmail.com>
[Dan: remove from pacman.conf and pacman.conf.5]
Signed-off-by: Dan McGee <dan@archlinux.org>
This allows a frontend to define its own download algorithm so that the
libfetch dependency can be omitted without using an external process.
The callback will be used when if it is defined, otherwise the old
behavior applies.
Signed-off-by: Sebastian Nowicki <sebnow@gmail.com>
[Dan: minor cleanups]
Signed-off-by: Dan McGee <dan@archlinux.org>
If the user switches from unstable repo to a stable one, it is quite hard to
sync its system with the new repo (the user will see many "Local is newer
than stable" messages, nothing more). That's why I introduced -Suu, which
treats a sync package like an upgrade, iff the package version doesn't match
with the local one's.
I added a new pactest (sync104.py) to test this, and I updated the
documentation of -Su.
Signed-off-by: Nagy Gabor <ngaba@bibl.u-szeged.hu>
[Dan: slight doc reword]
Signed-off-by: Dan McGee <dan@archlinux.org>
This fixes FS#14899. When running an -Sp operation without servers
configured for a repository, we would segfault, so add an assert to the
backend method returning the first server preventing a null pointer
dereference.
In addition, add a new error code to libalpm that indicates we have no
servers configured for a repository. This makes -Sy and -S <package>
operations fail gracefully and helpfully when a repo is set up with no
servers, as the default mirrorlist in Arch is provided this way.
Signed-off-by: Dan McGee <dan@archlinux.org>
This flag indicates that the front-end will not call alpm_trans_commit(),
so the database needn't be locked. This is the first step toward fixing
FS#8905.
If this flag is set, alpm_trans_commit() does nothing and returns with
an error (that has new code: PM_ERR_TRANS_NOT_LOCKED).
Signed-off-by: Nagy Gabor <ngaba@bibl.u-szeged.hu>
Signed-off-by: Dan McGee <dan@archlinux.org>
pmsyncpkg_t data sructure was removed:
1. pmpkg_t.reason is used instead of pmsyncpkg_t.newreason. (The target
packages come from sync repos, so we can use this field without any
problems. Upgrade transaction also uses this field to store this info.)
2. pmsyncpkg_t.removes was moved to pmpkg_t.removes.
This step requires careful programming, because we don't duplicate packages
when we add them to trans->packages. So we modify sync pkgcache when we
add this transaction-only info to our package. Hence it is important to
free this list when we remove any package from the target list
(remove_unresolvable, remove_conflicts, trans_free), otherwise this could
confuse the new sync transactions (with non-pacman GUI).
Overall, our code became ~100 line shorter, and we can call our helper
functions directly on trans->packages in sync.c, we don't need to maintain
parallel package lists.
Signed-off-by: Nagy Gabor <ngaba@bibl.u-szeged.hu>
Signed-off-by: Dan McGee <dan@archlinux.org>
This patch fixes FS#12059.
Now sync_addtarget can return with PM_ERR_PKG_IGNORED, which indicates that
although the requested package was found it is in ignorepkg, so alpm could
not add it to the transaction. So the front-end can decide what to do.
Signed-off-by: Nagy Gabor <ngaba@bibl.u-szeged.hu>
Signed-off-by: Dan McGee <dan@archlinux.org>
The from_md5 and to_md5 fields were a nice extra safety, which would avoid
trying to apply deltas on corrupted package files. However, they are not
strictly necessary, since xdelta should be able to detect that on its own.
The main problem is that it is impossible to compute these informations from
the delta only. So repo-add would not be able to compute the delta entry
based on just the delta file.
Signed-off-by: Xavier Chantry <shiningxc@gmail.com>
Signed-off-by: Dan McGee <dan@archlinux.org>
Enabled a new prompt to ask the user if they'd like to remove
unresolvable packages from the transaction rather than failing it.
Many pactest tests that used to fail now return success codes, because
pacman now issues a prompt allowing the user to cancel rather than
failing many transactions, and the pactest scripts always choose to
cancel with no error rather than failing. The only net effect is that
the return status of pacman is now 0 in cases where it used to be
nonzero.
Signed-off-by: Bryan Ischo <bryan@ischo.com>
Signed-off-by: Dan McGee <dan@archlinux.org>
This patch introduces the following function name convention:
_compute_ in function name: the return value must be freed.
_get_ in function name: the return value must not be freed.
Signed-off-by: Nagy Gabor <ngaba@bibl.u-szeged.hu>
Signed-off-by: Dan McGee <dan@archlinux.org>
The HoldPkg feature is even more important when the packages to be held are
pulled automatically by pacman, in a -Rc and -Rs operation. Before, it only
applied when the packages were explicitly requested by the user to be
removed. This patch extends holdpkg to -Rc and -Rs by doing the HoldPkg
check just before trans_commit.
Additionally, the whole HoldPkg stuff was moved to the front-end.
I changed the default behavior to "don't remove", so I modified remove030.py
pactest as well.
See also: FS#9173.
Original-work-by: Xavier Chantry <shiningxc@gmail.com>
Signed-off-by: Nagy Gabor <ngaba@bibl.u-szeged.hu>
Signed-off-by: Dan McGee <dan@archlinux.org>
Use libfetch naming in the code in place of libdownload names. This is in
preparation for dropping support for libdownload at some point as libfetch
can run on Linux.
Signed-off-by: Dan McGee <dan@archlinux.org>
From now on -Qu is an "outdated package" filter on local database.
(This is a behaviour change.)
This patch fixes some memleaks and makes the code cleaner, for details see
my comment on FS#7884.
FS#11868 is implemented.
Signed-off-by: Nagy Gabor <ngaba@bibl.u-szeged.hu>
Signed-off-by: Dan McGee <dan@archlinux.org>
This is more consistent with the private functions :
_alpm_db_get_{pkg,grp}cache
Signed-off-by: Xavier Chantry <shiningxc@gmail.com>
Signed-off-by: Dan McGee <dan@archlinux.org>
These two functions now take directly a package list rather than a database.
checkdbconflicts was renamed to checkconflicts.
Signed-off-by: Xavier Chantry <shiningxc@gmail.com>
This patch kills one of our hackish pseudo transactions: PRINTURIS.
(The other one is -Sw)
From now on, front-end must not call trans_commit in case of -Sp,
it should print the uris of target packages "by hand" instead.
PRINTURIS flag was removed, NOCONFLICTS flag can be passed to skip
conflict checks.
Signed-off-by: Nagy Gabor <ngaba@bibl.u-szeged.hu>
Signed-off-by: Dan McGee <dan@archlinux.org>
This function returns with the origin database of a package.
Signed-off-by: Nagy Gabor <ngaba@bibl.u-szeged.hu>
Signed-off-by: Dan McGee <dan@archlinux.org>
* Change the return values to be more informative.
It was previously boolean, only indicating if a sync package was newer than
a local package.
Now it is a simple wrapper to vercmp, handling the force flag.
* Remove the verbose output from _alpm_pkg_compare_versions.
The "force" message is not so useful.
The "package : local (v1) is newer than repo (v2)" message can be moved to
-Su operation.
For the -S operation, it is better to have something like :
"downgrading package from v1 to v2"
* Don't display the "up to date -- skipping" and "up to date -- reinstalling"
messages, when the local version is newer than the sync one.
* Fix the behavior of --needed option to not skip a target when the local
version is newer, and clarify its description.
* Add a new alpm_pkg_has_force function
This allows us to access the pkg->force field like any other package fields.
Signed-off-by: Xavier Chantry <shiningxc@gmail.com>
Signed-off-by: Dan McGee <dan@archlinux.org>
Now '-S provision' handling is done in the back-end.
In case of multiple providers, the first one is selected (behavior change:
deleted provision002.py). The old processing order was: literal, group,
provision; the new one: literal, provision, group. This is more rational,
but "pacman -S group" will be slower now. "pacman -S repo/provision" also
works. Provision was generalized to dependencies, so you can resolve deps by
hand: "pacman -S 'bash>2.0'" or "pacman -S 'core/bash>2.0'" etc. This can be
useful in makepkg dependency resolving. The changes were documented in
pacman manual.
alpm_find_pkg_satisfiers and _alpm_find_dep_satisfiers functions were
removed, since they are no longer needed.
I added some verbosity to "select provider instead of literal" and
"fallback to group".
Signed-off-by: Nagy Gabor <ngaba@bibl.u-szeged.hu>
Signed-off-by: Dan McGee <dan@archlinux.org>
Add a new totaldlcb callback function to libalpm and make pacman utilize it
when the TotalDownload option is enabled. This callback function is pretty
simple- it is meant to be called once at the beginning of a "list download"
action, and once at the end (with value 0 to indicate the list has been
finished). The frontend is responsible for keeping track of adding
individual file download amounts to the total xfered amount in order to
display some sort of overall progress.
Signed-off-by: Dan McGee <dan@archlinux.org>
We have been using unsigned long as a file size type for a while, which
works but isn't quite correct and could easily break. Worse was probably our
use of int in the download callback functions, which could be restrictive
for packages > 2GB in size.
Switch all file size variables to use off_t, which is the preferred type for
file sizes. Note that at least on Linux, all applications compiled against
libalpm must now be sure to use large file support, where _FILE_OFFSET_BITS
is defined to be 64 or there will be some weird issues that crop up.
Signed-off-by: Dan McGee <dan@archlinux.org>
This event was unused, was missing the equivalent EXTRACT_DONE event, and
was useless because we already have ADD / UPGRADE START and DONE events.
Signed-off-by: Xavier Chantry <shiningxc@gmail.com>
Signed-off-by: Dan McGee <dan@archlinux.org>
Actually, just rename _alpm_versioncmp to alpm_pkg_vercmp and get rid of the
need for a wrapper since it did nothing anyway.
Signed-off-by: Dan McGee <dan@archlinux.org>
* remove obsolete and unused *_cmp helper functions like deppkg_cmp and
_alpm_grp_cmp
* new alpm_list_remove_str function, used 6 times in handle.c
* remove _alpm_prov_cmp / _alpm_db_whatprovides and replace them by
a more general alpm_find_pkg_satisfiers with a cleaner implementation.
before: alpm_db_whatprovides(db, targ)
after: alpm_find_pkg_satisfiers(alpm_db_getpkgcache(db), targ)
* remove satisfycmp and replace alpm_list_find + satisfycmp usage by
_alpm_find_dep_satisfiers.
before : alpm_list_find(_alpm_db_get_pkgcache(db), dep, satisfycmp)
after : _alpm_find_dep_satisfiers(_alpm_db_get_pkgcache(db), dep)
* remove _alpm_pkgname_pkg_cmp, which was used with alpm_list_remove, and
use _alpm_pkg_find + alpm_list_remove with _alpm_pkg_cmp instead.
This commit actually get rids of all complicated and asymmetric _cmp
functions. I first thought these functions were worth it, be caused it
allowed us to reuse list_find and list_remove. But this was at the detriment
of the clarity and also the ease of use of these functions, dangerous
because of their asymmetricity.
Signed-off-by: Chantry Xavier <shiningxc@gmail.com>
Signed-off-by: Dan McGee <dan@archlinux.org>
It is hard to decipher what the transaction events actually notify you
of, and what parameters are passed to the callback function, without
looking at the code. This patch adds documentation for the _pmtransevt_t
enum in order to clarify what the event is for and what data is passed
when the callback is called.
Signed-off-by: Sebastian Nowicki <sebnow@gmail.com>
Signed-off-by: Chantry Xavier <shiningxc@gmail.com>
Signed-off-by: Dan McGee <dan@archlinux.org>
Using the graph structures that Nagy set up for dependency sorting, we now
do a similar process for deltas. Load up all of the deltas into a graph
object on which we can then apply Dijkstra's algorithm, using the new weight
field of graph struct.
We initialize the nodes weight using the base files that we can use in our
filecache (both filename and md5sum must match). The algorithm then picks
the best path among those that can be resolved.
Note that this algorithm has a few advantages over the old one:
1. It is completely file agnostic. These delta chains do not have to consist
of package files- this could be adopted to do delta-fied DBs.
2. It does not use the local_db anymore, or even care if a package or file
is currently installed. Instead, it only looks in the filecache for files
and packages that match delta chain entries.
Original-work-by: Dan McGee <dan@archlinux.org>
Signed-off-by: Chantry Xavier <shiningxc@gmail.com>
Start to move the delta struct away from an assumed package name scheme and
towards something that is package (or even filename) agnostic. This will
allow us much greater flexibility in the usage of deltas (maybe even sync
DBs some day) as well as allowing code outside of delta.h/delta.c to be much
cleaner with less of a need for snprintf() calls.
Signed-off-by: Dan McGee <dan@archlinux.org>
Add a new --disable-internal-download flag to configure allowing the
internal download code to be skipped. This will be helpful on platforms that
currently don't support either libdownload or libfetch (such as Cygwin) and
for just compiling a lighter weight pacman binary.
This was made really easy by our recent refactoring of the download code
into separate internal and external functions, as well as some error code
cleanup.
Signed-off-by: Dan McGee <dan@archlinux.org>
* -Rss removes all dependencies (including explicitly installed ones).
* updated documentation
* two pactest files added to test the difference between -Rs and -Rss
Signed-off-by: Nagy Gabor <ngaba@bibl.u-szeged.hu>
Its implementation was quite broken:
* add_loadtarget() might have silently filtered out some targets when
replacing an older version.
* This was used in sync.c to determine whether a target is implicit or not,
which is incorrect behavior. Before this patch we silently removed user
confirmed replacements; now we always warn on a replacement.
* remove001.py behavior was quite odd in adding same target 5 times to the
target list, we can change this behavior to be a failure.
Signed-off-by: Nagy Gabor <ngaba@bibl.u-szeged.hu>
[Xav: changed remove001 pactest accordingly]
Signed-off-by: Chantry Xavier <shiningxc@gmail.com>
[Dan: rewrote commit message]
Signed-off-by: Dan McGee <dan@archlinux.org>
Now pacman frontend uses this function instead of the compile-time libalpm
version number.
Signed-off-by: Nagy Gabor <ngaba@bibl.u-szeged.hu>
[Dan: fix one more spot where LIB_VERSION was used]
Signed-off-by: Dan McGee <dan@archlinux.org>
test_delta_md5sum and test_pkg_md5sum were simple wrappers to test_md5sum,
and only used once, so not very useful. I removed them.
Also, test_md5sum and alpm_pkg_checkmd5sum functions were a bit duplicated,
so I refactored them with a new _alpm_test_md5sum function in libalpm/util.c
Signed-off-by: Chantry Xavier <shiningxc@gmail.com>
I screwed up originally when I accepted the TotalDownload patch,
8ec27835f4. I didn't realize how deeply it
modified libalpm and I probably shouldn't have let it do what it did. This
commit reverts much of what that patch added in order to clean up our
internal function calls. We can find another way to do it right down the
road here but for now it has to go.
Signed-off-by: Dan McGee <dan@archlinux.org>
Remove what was a pretty weird abstraction in the libalpm backend. Instead
of parsing server URLs as we get them (of which we don't usually use more
than a handful anyway), wait until they are actually used, which allows us
to store them as a simple string list instead. This allows us to remove a
lot of code, and will greatly simplify the continuing refactoring of the
download code.
Signed-off-by: Dan McGee <dan@archlinux.org>
