Basically I'm the idiot that thought I could make it better and completely
forgot how freeing the contents of the original lists would screw up our
nice little diff extraction lists. This caused segfaults among other
problems. Last time I try to do that...
Program received signal SIGSEGV, Segmentation fault.
0x00007ffff627ce26 in strcmp () from /lib/libc.so.6
(gdb) bt
Signed-off-by: Dan McGee <dan@archlinux.org>
Follow-up to the previous "Only extract new DB entries" patch; move the
partial extraction code inside one side of the loop so we can use the same
code for actually doing file extraction.
Signed-off-by: Dan McGee <dan@archlinux.org>
This implements FS#15198. The idea apparently came from Csaba Henk
<csaba-ml <at> creo.hu> which submitted a patch to Frugalware, so thanks to
him, even though I did not look at the code :)
The idea is to only extract folders for new packages into the package
database and clean up the old directories. This is essentially implementing
Xyne's "rebase" script within pacman.
If using -Syy, just remove and extract everything.
If using -Sy :
1. Generate list of directories in DB
2. Generate list of directories in archive
3. Compare both
4. Clean up old directories
5. Extract new directories
Original-work-by: Allan McRae <allan@archlinux.org>
Signed-off-by: Xavier Chantry <shiningxc@gmail.com>
[Dan: fix compile error, s/int/size_t/]
Signed-off-by: Dan McGee <dan@archlinux.org>
Some users reported duplicated database entries in /var/lib/pacman/local/,
for example, both foo-1.0-1 and foo-2.0-1 subdirectories existed. (Bogus
3rd-party scripts, backup?) In this case pacman reported no error and its
behaviour was mysterious.
From now on, pacman detects this situation and prints an error message.
Signed-off-by: Nagy Gabor <ngaba@bibl.u-szeged.hu>
Signed-off-by: Dan McGee <dan@archlinux.org>
Sorry for this being such a huge patch, but I believe it is necessary for
quite a few reasons which I will attempt to explain herein. I've been
mulling this over for a while, but wasn't super happy with making the
download interface more complex. Instead, if we carefully order things in
the internal download code, we can actually make the interface simpler.
1. FS#15657 - This involves `name.db.tar.gz.part` files being left around the
filesystem, and then causing all sorts of issues when someone attempts to
rerun the operation they canceled. We need to ensure that if we resume a
download, we are resuming it on exactly the same file; if we cannot be
almost postive of that then we need to start over.
2. http://www.mail-archive.com/pacman-dev@archlinux.org/msg03536.html - Here
we have a lighttpd bug to ruin the day. If we send both a Range: header and
If-Modified-Since: header across the wire in a GET request, lighttpd doesn't
do what we want in several cases. If the file hadn't been modified, it
returns a '304 Not Modified' instead of a '206 Partial Content'. We need to
do a stat (e.g. HEAD in HTTP terms) operation here, and the proceed
accordingly based off the values we get back from it.
3. The mtime stuff was rather ugly, and relied on the called function to
write back to a passed in reference, which isn't the greatest. Instead, use
the power of the filesystem to contain this info. Every file downloaded
internally is now carefully timestamped with the remote file time. This
should allow the resume logic to work. In order to guarantee this, we need
to implement a signal handler that catches interrupts, notifies the running
code, and causes it to set the mtimes on the file. It then rethrows the
signal so the pacman signal handler (or any frontend) works as expected.
4. We did a lot of funky stuff in trying to track the DB last modified time.
It is a lot easier to just keep the downloaded DB file around and track the
time on that rather than in a funky dot file. It also kills a lot of code.
5. For GPG verification of the databases down the road, we are going to need
the DB file around for at least a short bit of time anyway, so this gets us
closer to that.
Signed-off-by: Dan McGee <dan@archlinux.org>
[Xav: fixed printf with off_t]
Signed-off-by: Xavier Chantry <shiningxc@gmail.com>
Pacman's fgets function in the API used hardcoded numbers to identify the size.
This is not good practice, so replace them with sizeof handling.
Signed-off-by: Laszlo Papp <djszapi@archlinux.us>
Signed-off-by: Dan McGee <dan@archlinux.org>
Commit 34e1413d75 attempted to implement lazy loading of package databases.
Although it took care of my main complaint (creating the database directory
if it didn't exist), it didn't allow sync repos to be registered before
alpm_option_set_dbpath() had been called.
With this patch, we no longer compute the individual repository DB paths
until necessary, allowing full lazy loading to work as intended, and
allowing us to drop the extra setlibpath() calls from the frontend. This
allows the changes introduced in a2cd48960 (but later reverted) to be added
back in again.
Signed-off-by: Dan McGee <dan@archlinux.org>
Add more untranslated strings, improve consistency, etc.
Signed-off-by: Xavier Chantry <shiningxc@gmail.com>
Signed-off-by: Dan McGee <dan@archlinux.org>
This basically started with this change :
/* Transaction */
struct __pmtrans_t {
- pmtranstype_t type;
pmtransflag_t flags;
pmtransstate_t state;
- alpm_list_t *packages; /* list of (pmpkg_t *) */
+ alpm_list_t *add; /* list of (pmpkg_t *) */
+ alpm_list_t *remove; /* list of (pmpkg_t *) */
And then I have to modify all the code accordingly.
This adds alpm_db_update() to the alpm_databases Doxygen group. The function
is described in more detail and a code example is given.
Signed-off-by: Dan McGee <dan@archlinux.org>
We don't need to create the directories when local or sync dbs are
registered. For example, if a sync db does not exist, we cannot even do
"pacman -Q" as an user.
Instead, we can create the local db if needed during the db_prepare
operation, and sync dbs on db_update.
Also remove some more useless abstractions in db_update and switch to a much
more efficient way to remove a sync db : rm -rf.
Signed-off-by: Xavier Chantry <shiningxc@gmail.com>
These db_open and db_close looked quite useless. And they caused the db
directory to be opened on a simple registering of a database. This is
totally unneeded, this opening can be delayed to when we actually need it.
Signed-off-by: Xavier Chantry <shiningxc@gmail.com>
This patch introduces the following function name convention:
_compute_ in function name: the return value must be freed.
_get_ in function name: the return value must not be freed.
Signed-off-by: Nagy Gabor <ngaba@bibl.u-szeged.hu>
Signed-off-by: Dan McGee <dan@archlinux.org>
Changelogs and install files were getting extracted into the local
db folder before it was manually created. This created issues for
uses with 0077 umasks and was highlighted with the new sudo handling
of umasks (FS#12263).
This moves the local db creation to its own function which is called
before the start of package archive extraction. Also, added a check
that the folder is actually created.
Signed-off-by: Allan McRae <allan@archlinux.org>
[Dan: rename to _alpm_db_prepare()]
Signed-off-by: Dan McGee <dan@archlinux.org>
If the delta line doesn't match our regex, we won't go and process it,
possibly walking off the end of the string.
Signed-off-by: Dan McGee <dan@archlinux.org>
We were using the stat() system call in quite a few places when we didn't
actually need anything the stat struct returned- we were simply checking for
file existence. access() will be more efficient in those cases.
Before (strace pacman -Ss pacman):
% time seconds usecs/call calls errors syscall
------ ----------- ----------- --------- --------- ----------------
33.16 0.005987 0 19016 stat64
After:
% time seconds usecs/call calls errors syscall
------ ----------- ----------- --------- --------- ----------------
34.85 0.003863 0 12633 1 access
7.95 0.000881 0 6391 7 stat64
Signed-off-by: Dan McGee <dan@archlinux.org>
Errors like the following one happen regularly (for unknown reasons...) :
error: could not open file /var/lib/pacman/local//glibc-2.7-9/depends: No
such file or directory
Anyway, every time an user reported an error like that, it always seemed
like he thought the error was caused by the double /, which is obviously
wrong.
Since db->path always include a trailing /, there is no need to add one when
concatenating paths in be_files.c or add.c.
Additionally, some static strings were switched to dynamic.
And the computation of the "dbpath"/"pkgname"-"pkgversion" was refactored
in db_read, db_write and db_remove with a get_pkgpath static function.
Signed-off-by: Xavier Chantry <shiningxc@gmail.com>
Signed-off-by: Dan McGee <dan@archlinux.org>
We have been using unsigned long as a file size type for a while, which
works but isn't quite correct and could easily break. Worse was probably our
use of int in the download callback functions, which could be restrictive
for packages > 2GB in size.
Switch all file size variables to use off_t, which is the preferred type for
file sizes. Note that at least on Linux, all applications compiled against
libalpm must now be sure to use large file support, where _FILE_OFFSET_BITS
is defined to be 64 or there will be some weird issues that crop up.
Signed-off-by: Dan McGee <dan@archlinux.org>
repo-add and db_read both assume that REPLACES and FORCE fields are in the
desc file, so do that for db_write as well (instead of depends file).
Note that db_write is currently only used on the local database. And the
only purpose of replaces and force in local database is for information
purpose (available on -Qi operations). So this is not a big problem.
Ref: http://www.archlinux.org/pipermail/pacman-dev/2008-May/011859.html
Acked-by: Dan McGee <dan@archlinux.org>
Signed-off-by: Xavier Chantry <shiningxc@gmail.com>
Signed-off-by: Dan McGee <dan@archlinux.org>
Name and version are computed from "/var/lib/pacman/..." pathname. And the
%NAME% and %VERSION% fields from the desc file were not even read. So now,
when we read the desc file, we make sure the %NAME% and %VERSION% fields are
consistent.
Signed-off-by: Nagy Gabor <ngaba@bibl.u-szeged.hu>
Signed-off-by: Xavier Chantry <shiningxc@gmail.com>
Signed-off-by: Dan McGee <dan@archlinux.org>
Any real call of this function doesn't specify a name or version ahead of
time, so just kill that functionality off. Now to remove those dummy
packages...
Signed-off-by: Dan McGee <dan@archlinux.org>
Commit 0460038447 caused a regression when
rereading the pkgcache after updating the on-disk databases. A rewinddir
call was errantly removed.
Instead of replacing the call to rewindir, clean up this whole mess.
db_scan is used only once and with target == NULL so there was actually half
the code of db_scan which was unused. This is gone now and replaced by a
single new db_populate function.
Dan: add_sorted ended up being 3x slower than one msort at the end, so I
changed back to that. I also made one pointer variable const and merged this
whole patch with my original fix for the rewinddir issue.
Signed-off-by: Xavier Chantry <shiningxc@gmail.com>
Signed-off-by: Dan McGee <dan@archlinux.org>
We have some useless abstractions like an alpm_db_rewind function. I've read
somewhere that readdir() was the worst filesystem function call invented,
and what do we do? Add a wrapper around it. Kill this abstraction and move
some other things into be_files that should be there anyway because they
are so tied to how a files backend works.
Signed-off-by: Dan McGee <dan@archlinux.org>
The deptest code (pacman -T) used by makepkg was mostly in the frontend.
There were 2 drawbacks:
1) the public splitdep function returns a pmdepend_t struct, but the
_alpm_dep_free function for freeing it is private. So there was a memleak.
2) there is a helper in the backend (satisfycmp in deps.c) which makes this
function much easier.
So this adds a new public alpm_deptest in libalpm/deps.c, which cleans
pacman_deptest in pacman/deptest.c a lot.
Besides, alpm_splitdep was made private, because the frontend no longer
requires it, and _alpm_dep_free is also private.
Finally the deptest001 pactest was extended.
Signed-off-by: Chantry Xavier <shiningxc@gmail.com>
Signed-off-by: Dan McGee <dan@archlinux.org>
Another elimination of a static length structure in libalpm. Should result
in a little more memory saved during execution of packages with lots of
deltas attached.
Signed-off-by: Dan McGee <dan@archlinux.org>
This also affects all structures with static strings, such as depmiss,
conflict, etc. This should help a lot with memory usage, and hopefully make
things a bit more "idiot proof".
Currently our pactest pass/fail rate is identical before and after this
patch. This is not to say it is a perfect patch- I have yet to pull valgrind
out. However, this should be quite safe to use in all situations from here
on out, and we can start plugging the memleaks.
Original-work-by: Aaron Griffin <aaronmgriffin@gmail.com>
Signed-off-by: Dan McGee <dan@archlinux.org>
These used FUNCTION output level and not DEBUG, so I didn't catch them way
back when I removed those gettext calls. Remove them now (which exposed a
nice little memory access error elsewhere in the code). This should have a
slight speedup effect on the code too as we no longer have to make the
gettext call even when these messages aren't printed.
Signed-off-by: Dan McGee <dan@archlinux.org>
Update the GPL boilerplate to direct people to the GNU website for a copy of
the license, as well as bump all of Judd's copyrights to 2007.
Signed-off-by: Dan McGee <dan@archlinux.org>
Legacy code is hitting the trash here. Remove unnecessary _alpm_time2string
time storage abstraction in favor of just writing the time_t value to the
disk.
The only drawback is that everyone's sync DBs will have to be updated at
least once so that the lastupdate values are stored right. :)
Signed-off-by: Dan McGee <dan@archlinux.org>
Holy inefficient batman! For a pacman -Qt operation (when we are using
compute_requiredby and not database entries), splitdep was being called ~1.3
million times on my local database. By splitting when we read the DB, we
drop this number to around 1700 and save a LOT of time in doing so (a 5x
increase in pacman -Qt speed here).
Note that the depends alpm_list_t in the package struct is no longer a
string list, but a list of pmdepent_t objects.
Signed-off-by: Dan McGee <dan@archlinux.org>
Instead of using the often-busted REQUIREDBY entries in the pacman database,
compute them each time they are required. This should help many things:
1. Simplify the codebase
2. Prevent future database corruption
3. Ensure when we do use requiredby, it is always correct
4. Shrink the pmpkg_t memory overhead
Signed-off-by: Dan McGee <dan@archlinux.org>
Too many fields were being shown on -Qip output, and sizes were not always
correct (-Qi and -Qip output on the same package did not agree).
Signed-off-by: Dan McGee <dan@archlinux.org>
Commit 47622eef4d introduced localized times
in the metadata by way of storing the UNIX epoch value instead of a hard
coded date string. However, it missed a few things:
* If we weren't in the C/POSIX/en_US locale, the date parsing would fail
as it tried to use the abbreviations of the locale being used. Fix this
by switching the LC_TIME value before we parse a date.
* We used ctime to print the date value, which is always the C locale
string. Instead, use strftime to print a localized date string.
Signed-off-by: Dan McGee <dan@archlinux.org>
The recommended C99 way to print the value of a time_t is to cast it to
uintmax_t or intmax_t. Do this to ensure compatability with all platforms.
Signed-off-by: Dan McGee <dan@archlinux.org>
The code for setting the lastupdate file path still referenced both the
root and dbpath, when we should only be using dbpath. Fix it.
Signed-off-by: Dan McGee <dan@archlinux.org>
Packages and DBs now support using the UNIX epoch (seconds since Jan 1, 1970)
for use in builddate and installdate. This will only affect newly built
packages. Old existing packages with the text format are still supported, but
this is deprecated.
In the case of removal of text time support, this code will fail gracefully,
returning the start of the epoch for broken packages.
Signed-off-by: Aaron Griffin <aaronmgriffin@gmail.com>
