We were using atol(), which on 32 bit, cannot handle values greater than
2GiB, which is fail.
Switch to a strtoull() wrapper function tailored toward parsing off_t
values. This allows parsing of very large positive integer values. off_t
is a signed type, but in our usages, we never parse or have a need for
negative values, so the function will return -1 on error.
Before:
$ pacman -Si flightgear-data | grep Size
Download Size : 2097152.00 K
Installed Size : 2097152.00 K
After:
$ ./src/pacman/pacman -Si flightgear-data | grep Size
Download Size : 2312592.52 KiB
Installed Size : 5402896.00 KiB
Signed-off-by: Dan McGee <dan@archlinux.org>
Free "md5sum" if md5_file() fails in alpm_compute_md5sum(). Free
"sha256sum" if sha2_file() fails in alpm_compute_sha256sum().
Signed-off-by: Lukas Fleischer <archlinux@cryptocrack.de>
Signed-off-by: Dan McGee <dan@archlinux.org>
There were many cases where the string coming in was a blank line, e.g.
"\n\0", length 1. The trim routine starts by trimming leading spaces,
thus trimming everything. We would then proceed to do a memmove of the
NULL byte, which is completely worthless as we can just assign it
instead.
Signed-off-by: Dan McGee <dan@archlinux.org>
This is a refactor and refresh of the code used to find where we should
download packages.
* Incorporate suggestions from FS#25435 to use TMPDIR from the
environment if set, otherwise fall back to /tmp as before.
* Make the writability tests a bit more in depth. We now do a three part
check consisting of:
- S_ISDIR(): is this even a directory
- access(W_OK): is this directory writable by the current user.
Unfortunately for root, this almost always returns that it is, but
in the case of a RO mount or NFS share inaccessible to root, this
check will exclude the directory.
- mode & (any write bit): is there a writable bit set on this
directory. This makes it possible to enforce a read-only cache
directory by setting permissions to 0555, for example.
Signed-off-by: Dan McGee <dan@archlinux.org>
If the string was zero-length to begin with, or consists of only newline
characters, nothing stopped us from incrementing right off the front of
the string. Ensure len stays above zero the whole time.
Signed-off-by: Dan McGee <dan@archlinux.org>
This is standard procedure elsewhere and cuts down on translations that
won't be seen (and we don't want if we need English debug output
anyway).
Signed-off-by: Dan McGee <dan@archlinux.org>
We don't write with extra or unknown whitespace, so there is little
reason for us to trim it when reading either. This also fixes the
hopefully never encountered "paths that start or end with spaces" issue,
for which two pactests have been added. The tests also contain other
evil characters that we have encountered before and handle just fine,
but it doesn't hurt to ensure we don't break such support in the future.
Signed-off-by: Dan McGee <dan@archlinux.org>
This is a wrapper function for access() which logs some debug
information and eases handling in case of split directory and filename.
Signed-off-by: Florian Pritz <bluewind@xinu.at>
Signed-off-by: Dan McGee <dan@archlinux.org>
This addresses FS#25141. We shouldn't remove every empty directory we
come across during the removal process unless it is truly not known to
any other package. This will prevent removal of essential directories
such as '/var/lock/'.
This is accomplished by first checking the empty/non-empty status of a
directory, which was previously done implicitly by calling rmdir() and
ignoring errors. We do this to avoid the next (new) check in most cases,
which is to look at all local packages to see if the to-be-removed
directory is present in another packages' filelist. If we do not find it
anywhere, then we remove it, else we keep the file around.
The pactest has been updated to test more cases, as well as finding a
flaw in the original expected to fail case- we need separate DIR and
FILE based EXIST rules.
Signed-off-by: Dan McGee <dan@archlinux.org>
Modifying prefix caused tmp directories to be left behind after
running scriptlets, and the path '/' to be passed to _alpm_rmrf. Broken
in f01c6f.
Signed-off-by: Dave Reisner <dreisner@archlinux.org>
Signed-off-by: Dan McGee <dan@archlinux.org>
Only one of these looked like a real red flag, in find_requiredby(), but
it doesn't hurt to fix several of them up anyway.
Unfortunately, we can't turn this on universally due to things like the
sync(), remove(), etc. builtins which we often use as variable names.
Signed-off-by: Dan McGee <dan@archlinux.org>
We can reorganize things a bit to not require reading a directory-only
entry first (or at all). This was noticed while working on some pactest
improvements, but should be a good step forward anyway.
Also make _alpm_splitname() a bit more generic in where it stores the
data it parses.
Signed-off-by: Dan McGee <dan@archlinux.org>
Discovered this when doing some pactest rewrite work to generate
archives in memory only. If a sync database file or PKGINFO file is
missing a newline on the final line, the text from that line gets tossed
aside and never read into the package struct. This is pretty critical
when that last line is a depend or something.
Signed-off-by: Dan McGee <dan@archlinux.org>
This is the last user of our global handle object. Once again the diff
is large but the functional changes are not.
Signed-off-by: Dan McGee <dan@archlinux.org>
This requires a lot of line changes, but not many functional changes as
more often than not our handle variable is already available in some
fashion.
Signed-off-by: Dan McGee <dan@archlinux.org>
The vast majority of the time we will just be passing the same string
value on to the lstat() call. The only time we need to duplicate it is
if the path ends in '/'. In one run using a profiler, only 400 of the
200,000 calls (0.2%) required the string to be copied first.
Signed-off-by: Dan McGee <dan@archlinux.org>
This keeps duplicate code to a minimum. This will come in more handy as
we refactor some of these option setters away.
Signed-off-by: Dan McGee <dan@archlinux.org>
This allows us to not require the context (e.g. handle) when calling
this function. Also beef up the checks in the two callers of this
function to bail if the last return code is not ARCHIVE_EOF, which is
the expected value.
This requires a change to one of the pactest return codes and the
overall result of the test, but results in a much safer operating
condition whereby invalid database entries will stop the operation.
Signed-off-by: Dan McGee <dan@archlinux.org>
This will make the patching process less invasive as we start to remove
this variable from all source files.
Signed-off-by: Dan McGee <dan@archlinux.org>
The usefulness of this is rather limited due to it not being compiled
into production builds. When you do choose to see the output, it is
often overwhelming and not helpful. The best bet is to use a debugger
and/or well-placed fprintf() statements.
Signed-off-by: Dan McGee <dan@archlinux.org>
This does touch a lot of things, and hopefully doesn't break things on
other platforms, but allows us to also clean up a bunch of crud that no
longer needs to be there.
Signed-off-by: Dan McGee <dan@archlinux.org>
This is the standard, and we have had a few of these introduced lately
that should not be here.
Done with:
find -name '*.c' | xargs sed -i -e 's#if (#if(#g'
find -name '*.c' | xargs sed -i -e 's#while (#while(#g'
Signed-off-by: Dan McGee <dan@archlinux.org>
This is a rather serious data corruption issue that luckily manifested
itself today in a noticable way. A package in testing had replaces
entries read in as ["%RE pkgname", "%RE"] which was clearly wrong. This
happens when we hit the end of an archive block, do not have a newline,
and have to continue reading from the next block to complete the line.
Signed-off-by: Dan McGee <dan@archlinux.org>
This was discussed and more or less agreed upon on the mailing list. A
huge checkin, but if we just do it and let people adjust the pain will
end soon enough. Rebasing should be relatively straighforward for anyone
that sees conflicts; just be sure you use the new return style if
possible.
The following semantic patch was used to do the change, along with some
hand-massaging in order to preserve parenthesis where appropriate:
The semantic match that finds this problem is as follows, although some
hand-massaging was done in order to keep parenthesis where appropriate:
(http://coccinelle.lip6.fr/)
// <smpl>
@@
expression a;
@@
- return(a);
+ return a;
// </smpl>
A macros_file was also provided with the following content:
Additional steps taken, mainly for ASSERT() macros:
$ sed -i -e 's#return(NULL)#return NULL#' lib/libalpm/*.c
$ sed -i -e 's#return(-1)#return -1#' lib/libalpm/*.c
Signed-off-by: Dan McGee <dan@archlinux.org>
A lot of these were places that should have used the same message but
didn't, or were very easy to convert to using the same message and
letting some of the burden off of the translators.
Signed-off-by: Dan McGee <dan@archlinux.org>
We only call these from the transaction init and teardown, so move them
to that file, mark them static, and push more of the logic of handle
manipulation into these functions.
Signed-off-by: Dan McGee <dan@archlinux.org>
According to FOPEN(3), using fclose on an fdopen'd file stream also
closes the underlying file descriptor. This happened in _alpm_lckmk
(util.c), which meant that when alpm_trans_release closed it again, the
log file (which reused the original file descriptor) was closed instead.
Signed-off-by: Jonathan Conder <jonno.conder@gmail.com>
Signed-off-by: Dan McGee <dan@archlinux.org>
We located files in a few places but didn't check if they were files or
directories. Ensure they are actually files using stat() and S_ISREG(); this
showed itself when trying to download to the directory name itself in
FS#22645.
Signed-off-by: Dan McGee <dan@archlinux.org>
Noted in FS#22697. When I factored out _alpm_parsedate() into a common
function, I didn't move the <locale.h> include properly, causing a build
failure when NLS is disabled and this header isn't automatically included
everywhere.
Signed-off-by: Dan McGee <dan@archlinux.org>
We don't need to create a temporary copy of the string if we are smart with
our pointer manipulation and string copying. This saves a bunch of string
duplication during database parsing, both local and sync.
Signed-off-by: Dan McGee <dan@archlinux.org>
None of these warn at the normal "-Wall -Werror" level, but casts do occur
that we are fine with. Make them explicit to silence some warnings when
using "-Wconversion".
Signed-off-by: Dan McGee <dan@archlinux.org>
The old function was written in a time before we relied on it for nearly
every operation. Since then, we have switched to the archive backend and now
fast parsing is a big deal.
The former function made a per-character call to the libarchive
archive_read_data() function, which resulted in some 21 million calls in a
typical "load all sync dbs" operation. If we instead do some buffering of
our own and read the blocks directly, and then find our newlines from there,
we can cut out the multiple layers of overhead and go from archive to parsed
data much quicker.
Both users of the former function are switched over to the new signature,
made easier by the macros now in place in the sync backend parsing code.
Performance: for a `pacman -Su` (no upgrades available),
_alpm_archive_fgets() goes from being 29% of the total time to 12% The time
spent on the libarchive function being called dropped from 24% to 6%.
This pushes _alpm_pkg_find back to the title of slowest low-level function.
Signed-off-by: Dan McGee <dan@archlinux.org>
This is prepping for the addition of a hash field to each package to greatly
speed up the string comparisons we frequently do on package name in
_alpm_pkg_find.
Signed-off-by: Dan McGee <dan@archlinux.org>
Rather than hiding these warnings, show them to the user as they happen.
This will prevent things such as hiding full filesystem errors (ENOSPC) from
the user as seen in FS#11639.
Signed-off-by: Dan McGee <dan@archlinux.org>
[Allan: adjust warning wording and add gettext calls]
Signed-off-by: Allan McRae <allan@archlinux.org>
Signed-off-by: Dan McGee <dan@archlinux.org>
The splitname function is a general utility function and so is better
suited to util.h. Rename it to _alpm_splitname to indicate it is an
internal libalpm function as was the case prior to splitting local and
sync db handling.
Signed-off-by: Allan McRae <allan@archlinux.org>
I've noticed my Atom-powered laptop is dog-slow when doing integrity checks
on packages, and it turns out our MD5 implementation isn't near as good as
that provided by OpenSSL. Using their routines instead provided anywhere
from a 1.4x up to a 1.8x performance benefit over our built-in MD5 function.
This does not remove the MD5 code from our codebase, but it does enable
linking against OpenSSL to get their much faster implementation if it is
available on whatever platform you are using. At configure-time, we will
default to using it if it is available, but this can be easily changed by
using the `--with-openssl` or `--without-openssl` arguments to configure.
Signed-off-by: Dan McGee <dan@archlinux.org>
Fixes FS#18770, and hopefully an occasional deadlock in my frontend as well.
For simplicity it redirects all scriptlet output through SCRIPTLET_INFO, and
all callbacks in the child process have been replaced for thread-safety.
Signed-off-by: Jonathan Conder <j@skurvy.no-ip.org>
Signed-off-by: Dan McGee <dan@archlinux.org>
Prevents compiler warnings when building with -D_FORTIFY_SOURCE=2
Signed-off-by: Allan McRae <allan@archlinux.org>
Signed-off-by: Dan McGee <dan@archlinux.org>
Check that writing to destination file actually occurs in
_alpm_copyfile. Required adding a new error (PM_ERR_WRITE)
as none of the others appeared appropriate.
Prevents compiler warning when using -D_FORTIFY_SOURCE=2.
Signed-off-by: Allan McRae <allan@archlinux.org>
Signed-off-by: Dan McGee <dan@archlinux.org>
Follow the HACKING guidelines and always use != 0 or == 0 rather
than negation within conditional statements to improve clarity.
Most of these are !strcmp usages which is the example of what not
to do in the HACKING document.
Signed-off-by: Allan McRae <allan@archlinux.org>
This implements FS#15198. The idea apparently came from Csaba Henk
<csaba-ml <at> creo.hu> which submitted a patch to Frugalware, so thanks to
him, even though I did not look at the code :)
The idea is to only extract folders for new packages into the package
database and clean up the old directories. This is essentially implementing
Xyne's "rebase" script within pacman.
If using -Syy, just remove and extract everything.
If using -Sy :
1. Generate list of directories in DB
2. Generate list of directories in archive
3. Compare both
4. Clean up old directories
5. Extract new directories
Original-work-by: Allan McRae <allan@archlinux.org>
Signed-off-by: Xavier Chantry <shiningxc@gmail.com>
[Dan: fix compile error, s/int/size_t/]
Signed-off-by: Dan McGee <dan@archlinux.org>
* It makes the code clearer to read/understand
* Cppcheck tool doesn't show this anymore: [./util.c:215]: (error) Resource leak: fd
[Dan: don't change the coding style]
Signed-off-by: Dan McGee <dan@archlinux.org>
Thanks to Laszlo Papp <djszapi@archlinux.us> for the following catch:
opendir(path)) == (DIR *)-1;
is maybe the result of misunderstanding the manpage. If an opendir() call
isn't successful it returns NULL rather than '(DIR *)-1'.
Noticed-by: Laszlo Papp <djszapi@archlinux.us>
Signed-off-by: Dan McGee <dan@archlinux.org>
After our recent screwup with size_t and ssize_t in the download code, I
found the `-Wsign-conversion` flag to GCC to see if we were doing anything
else boneheaded. I didn't find anything quite as bad, but we did have some
goofups- most of our public unsigned methods would return -1 on error, which
is a bit odd in an unsigned context.
Signed-off-by: Dan McGee <dan@archlinux.org>
This function is unused since commit
358cc5804a.
Signed-off-by: Xavier Chantry <shiningxc@gmail.com>
[Dan: also kill from util.h]
Signed-off-by: Dan McGee <dan@archlinux.org>
If /sbin is not in the PATH and sudo is used, ldconfig cannot be found. So
use /sbin/ldconfig instead. The code checked for the existence of
/sbin/ldconfig anyway..
Signed-off-by: Marc - A. Dahlhaus <mad@wol.de>
Signed-off-by: Xavier Chantry <shiningxc@gmail.com>
Signed-off-by: Dan McGee <dan@archlinux.org>
This fixes FS#15294.
The code to run a command inside a chroot was refactored from the
_alpm_runscriptlet function to _alpm_run_chroot.
Signed-off-by: Xavier Chantry <shiningxc@gmail.com>
Instead of appending the prefix to each entry name, we can chdir to the
prefix before extracting, and restoring when it is done.
This seems to work better with the strange and special case of FS#12148
where an archive contained the "./" entry.
Signed-off-by: Xavier Chantry <shiningxc@gmail.com>
This patch fixes FS#12148 ('unstable' regular file).
I also changed the other archive_entry_set_mode usage in add.c to
archive_entry_set_perm.
Since I cannot find any relevant info in libarchive manual, I quote
Tim Kientzle (the author of libarchive) here, and I say thank you for
his help.
*** Tim Kientzle wrote *************************************
This is the problem in libalpm/util.c:
323 if(S_ISREG(st->st_mode)) {
324 archive_entry_set_mode(entry, 0644);
325 } else if(S_ISDIR(st->st_mode)) {
326 archive_entry_set_mode(entry, 0755);
327 }
Your example unstable.db.tar.gz is not empty. It has
one entry in it, called "./". That entry is marked
as a directory. But, when you call archive_entry_set_mode(),
you are changing the file type! archive_read_extract()
then creates the file /var/unstable as you requested.
(archive_read_extract() will replace an empty directory
with a file.)
You should either set the mode value correctly:
323 if(S_ISREG(st->st_mode)) {
324 archive_entry_set_mode(entry, IFREG | 0644);
325 } else if(S_ISDIR(st->st_mode)) {
326 archive_entry_set_mode(entry, IFDIR | 0755);
327 }
Or use archive_entry_set_perm(), which does not change
the file type:
323 if(S_ISREG(st->st_mode)) {
324 archive_entry_set_perm(entry, 0644);
325 } else if(S_ISDIR(st->st_mode)) {
326 archive_entry_set_perm(entry, 0755);
327 }
************************************************************
Signed-off-by: Nagy Gabor <ngaba@bibl.u-szeged.hu>
Signed-off-by: Dan McGee <dan@archlinux.org>
This patch introduces the following function name convention:
_compute_ in function name: the return value must be freed.
_get_ in function name: the return value must not be freed.
Signed-off-by: Nagy Gabor <ngaba@bibl.u-szeged.hu>
Signed-off-by: Dan McGee <dan@archlinux.org>
This is the first step in being able to automatically remove phantom
lock files.
Signed-off-by: Allan McRae <allan@archlinux.org>
[Dan: fix compilation warnings]
Signed-off-by: Dan McGee <dan@archlinux.org>
It is possible to throw EINTR from a system call such as open(), close(), or
waitpid() if custom signal handlers are set up and they are not initialized
with the SA_RESTART flag. This was noticed by Andreas Radke when ^C (SIGINT)
was given during the call to waitpid(), causing it to throw the EINTR error
and we could not accommodate it.
Simply wrap these calls in a simple loop that allows us to retry the call if
interrupted.
Signed-off-by: Dan McGee <dan@archlinux.org>
Probably a tweakable "lockdb-retry" option was planned which is not
implemented. (Now it should be implemented in front-end.)
So now this variable was unused and caused a small memleak.
(FREE(dir) was not reached in case of error.)
Signed-off-by: Nagy Gabor <ngaba@bibl.u-szeged.hu>
Signed-off-by: Dan McGee <dan@archlinux.org>
Due to differences in handling va_list between i686 and x86_64, this bug
can only be seen on x86_64. va_list usage is not allowed but we had been
getting away with it. See
http://lists.opensuse.org/opensuse-programming/2008-02/msg00005.html
for details and explanation.
This fixes FS#11096.
Signed-off-by: Dan McGee <dan@archlinux.org>
We were using the stat() system call in quite a few places when we didn't
actually need anything the stat struct returned- we were simply checking for
file existence. access() will be more efficient in those cases.
Before (strace pacman -Ss pacman):
% time seconds usecs/call calls errors syscall
------ ----------- ----------- --------- --------- ----------------
33.16 0.005987 0 19016 stat64
After:
% time seconds usecs/call calls errors syscall
------ ----------- ----------- --------- --------- ----------------
34.85 0.003863 0 12633 1 access
7.95 0.000881 0 6391 7 stat64
Signed-off-by: Dan McGee <dan@archlinux.org>
The start of a few commits to remove some PATH_MAX usage from our code. Use
a dynamically allocated string instead.
Signed-off-by: Dan McGee <dan@archlinux.org>
Rework to use a single #define for the buffsize, and in the process clean up
some other code and double the default buffer size.
Signed-off-by: Dan McGee <dan@archlinux.org>
Using the graph structures that Nagy set up for dependency sorting, we now
do a similar process for deltas. Load up all of the deltas into a graph
object on which we can then apply Dijkstra's algorithm, using the new weight
field of graph struct.
We initialize the nodes weight using the base files that we can use in our
filecache (both filename and md5sum must match). The algorithm then picks
the best path among those that can be resolved.
Note that this algorithm has a few advantages over the old one:
1. It is completely file agnostic. These delta chains do not have to consist
of package files- this could be adopted to do delta-fied DBs.
2. It does not use the local_db anymore, or even care if a package or file
is currently installed. Instead, it only looks in the filecache for files
and packages that match delta chain entries.
Original-work-by: Dan McGee <dan@archlinux.org>
Signed-off-by: Chantry Xavier <shiningxc@gmail.com>
This crude function allows reading from an archive on a line-by-line basis
similar to the familiar fgets() call on a FILE stream. This is the first
step in being able to read DB entries straight from an archive.
Signed-off-by: Dan McGee <dan@archlinux.org>
test_delta_md5sum and test_pkg_md5sum were simple wrappers to test_md5sum,
and only used once, so not very useful. I removed them.
Also, test_md5sum and alpm_pkg_checkmd5sum functions were a bit duplicated,
so I refactored them with a new _alpm_test_md5sum function in libalpm/util.c
Signed-off-by: Chantry Xavier <shiningxc@gmail.com>
Fix for FS#9176.
A previous commit (6e8daa553b) already forced all database files to 644.
Now the directories are also forced to 755.
Additionally, repo-add now sets the umask to 022, just like makepkg does, to
fix the problem at its root.
Signed-off-by: Chantry Xavier <shiningxc@gmail.com>
Signed-off-by: Dan McGee <dan@archlinux.org>
For example, if the cachedir is a broken symlink or a non writable
directory, pacman fallbacks to /tmp/. Just before doing that, it freed the
handle->cachedirs list twice !
once in _alpm_filecache_setup, and once in alpm_option_set_cachedirs. So the
first one was removed.
Fixes FS#9186.
Signed-off-by: Chantry Xavier <shiningxc@gmail.com>
By attempting to stat the cachedir when we load the pacman config, pacman
bails out if it is a non-existant directory, even if it will never be
needed. This is unfortunate as it is only used for sync transactions anyway.
Instead, wait until we need it in _alpm_filecache_setup to actually do
anything.
Reported as FS#9096.
Signed-off-by: Dan McGee <dan@archlinux.org>
