Commit Graph

162 Commits

Author SHA1 Message Date
Dave Reisner d6ccd44390 include config.h via Makefiles
Ensures that config.h is always ordered correctly (first) in the
includes. Also means that new source files get this for free without
having to remember to add it.

We opt for -imacros over -include as its more portable, and the
added constraint by -imacros doesn't bother us for config.h.

This also touches the HACKING file to remove the explicit mention of
config.h as part of the includes.

Signed-off-by: Dave Reisner <dreisner@archlinux.org>
Signed-off-by: Dan McGee <dan@archlinux.org>
2011-12-21 18:13:17 -06:00
Dan McGee dee8144ce5 Add helper method for creating and opening archive object
This moves the common setup code of about 5 different callers into one
method. Error messages will now be common and shared in all places;
several paths did not have any messages at all before.

In addition, we now pick an ideal block size for the archive read based
off the larger value of our default buffer size or the st.st_blksize
field. For a filesystem such as NFS, this is often much larger than the
default 8192- values such as 32768 and 131072 are common.

Signed-off-by: Dan McGee <dan@archlinux.org>
2011-11-16 14:51:06 -06:00
Dan McGee 6df558177f Convert package and database archive reads to use file descriptors
This gives us a bit more control and over the archive reading process,
and a bit less is done behind the scenes. It also allows us to use
fstat() in preference to stat(), which should avoid some potential race
conditions.

Some reorganization is necessary to move the stat calls after the open()
calls. Error handling and cleanup in general is also improved, as we had
several potential memory and file handle leaks before in some error
paths.

Signed-off-by: Dan McGee <dan@archlinux.org>
2011-11-01 10:27:31 -05:00
Dan McGee 3343185473 Introduce ALPM_BUFFER_SIZE constant
This takes the place of three previously used constants:
ARCHIVE_DEFAULT_BYTES_PER_BLOCK, BUFFER_SIZE, and CPBUFSIZE.

In libarchive 3.0, the first constant will be no more, so we can ensure
we are forward-compatible by removing our usage of it now. The rest are
unified for consistency.

By default, we will use the value of BUFSIZ provided by <stdio.h>, which
is 8192 on Linux. If that is undefined, a default value is provided.

Signed-off-by: Dan McGee <dan@archlinux.org>
2011-10-27 14:59:24 -05:00
Dan McGee e0acf2f144 Refactor download payload reset and free
This was done to squash a memory leak in the sync database download
code. When we downloaded a database and then reused the payload struct,
we could find ourselves calling get_fullpath() for the signatures and
overwriting non-freed values we had left over from the database
download.

Refactor the payload_free function into a payload_reset function that we
can call that does NOT free the payload itself, so we can reuse payload
structs. This also allows us to move the payload to the stack in some
call paths, relieving us of the need to alloc space.

Signed-off-by: Dan McGee <dan@archlinux.org>
2011-09-28 13:01:03 -05:00
Dan McGee d8fab9b441 be_sync: fetch only entry mode, not full stat struct
This saves a lot of unnecessary work since we don't need any of the
other fields in the stat struct.

Signed-off-by: Dan McGee <dan@archlinux.org>
2011-09-27 17:49:34 -05:00
Dan McGee c406949226 Ensure database validity flags are set in invalid signature case
We returned the right error code but never set the flags accordingly.
Also, now that we can bail early, ensure we set the error code.

Signed-off-by: Dan McGee <dan@archlinux.org>
2011-09-22 16:01:11 -05:00
Dan McGee ec790ced7c signing: add a process and retry loop for database signatures
In reality, there is no retrying that happens as of now because we don't
have any import or changing of the keyring going on, but the code is set
up so we can drop this in our new _alpm_process_siglist() function. Wire
up the basics to the sync database validation code, so we see something
like the following:

    $ pacman -Ss unknowntrust
    error: core: signature from "Dan McGee <dpmcgee@gmail.com>" is unknown trust
    error: core: signature from "Dan McGee <dpmcgee@gmail.com>" is unknown trust
    error: database 'core' is not valid (invalid or corrupted database (PGP signature))

    $ pacman -Ss missingsig
    error: core: missing required signature
    error: core: missing required signature
    error: database 'core' is not valid (invalid or corrupted database (PGP signature))

Yes, there is some double output, but this should be fixable in the
future.

Signed-off-by: Dan McGee <dan@archlinux.org>
2011-09-20 10:23:11 -05:00
Dan McGee 994cb4da4f Allow our PGP helper method to pass back the signature results
This will make its way up the call chain eventually to allow trusting
and importing of keys as necessary.

Signed-off-by: Dan McGee <dan@archlinux.org>
2011-09-20 10:23:11 -05:00
Dan McGee 83a1e4fee3 Clean up handling of size fields
We currently have csize, isize, and size concepts, and sometimes the
difference isn't clear. Ensure the following holds:

* size (aka csize): always the compressed size of the package; available
  for everything except local packages (where it will return 0)
* isize: always the installed size of the package; available for all
  three package types

Signed-off-by: Dan McGee <dan@archlinux.org>
2011-09-02 19:26:07 -05:00
Dan McGee 234b6ffc2c Parse > 2GiB file sizes correctly
We were using atol(), which on 32 bit, cannot handle values greater than
2GiB, which is fail.

Switch to a strtoull() wrapper function tailored toward parsing off_t
values. This allows parsing of very large positive integer values. off_t
is a signed type, but in our usages, we never parse or have a need for
negative values, so the function will return -1 on error.

Before:
    $ pacman -Si flightgear-data | grep Size
    Download Size  : 2097152.00 K
    Installed Size : 2097152.00 K

After:
    $ ./src/pacman/pacman -Si flightgear-data | grep Size
    Download Size  : 2312592.52 KiB
    Installed Size : 5402896.00 KiB

Signed-off-by: Dan McGee <dan@archlinux.org>
2011-08-29 19:57:05 -05:00
Dan McGee b221af660d Database read optimizations
Hard to believe there was still more room to improve on this, but I
found an easily correctable oversight tonight. Our databases (both sync
and local) contain many blank lines, and we were not moving onto the
next line right away in these cases; instead we would proceed through
our strcmp() conditional checks as normal.

Some local numbers follow to show the effects of this patch:

Sync `-Ss foobarbaz`:
71,709 blank lines skipped early
~1,505,889 strcmp() calls avoided (21 per line)
~15% speed improvement (.210 --> .179 sec)

Local `-Qs foobarbaz`:
6,823 blank lines skipped early
115,991 strcmp() calls avoided (17 per line)
~6% speed improvement (.080 -> .071 sec)

Signed-off-by: Dan McGee <dan@archlinux.org>
2011-08-28 23:49:27 -05:00
Lukas Fleischer 455ca55f4e be_sync.c: Fix memory leak in alpm_db_update()
Free "syncpath" and restore umask if we fail to grab a lock.

Signed-off-by: Lukas Fleischer <archlinux@cryptocrack.de>
Signed-off-by: Dan McGee <dan@archlinux.org>
2011-08-28 19:40:22 -05:00
Dan McGee e1dce078b2 Remove argument from check_pgp_helper
This one wasn't all that necessary as we only used it in one place in
the function, which can be checked easily enough at the call site.

Signed-off-by: Dan McGee <dan@archlinux.org>
2011-08-25 16:12:49 -05:00
Dan McGee d9e5dab6ac Remove alpm_option_get_* usage from backend
These are all available directly on the handle without indirection.

Signed-off-by: Dan McGee <dan@archlinux.org>
2011-08-19 19:12:21 -05:00
Dave Reisner 24824b54ce dload: add 'unlink_on_fail' to payload struct
Let callers of _alpm_download state whether we should delete on fail,
rather than inferring it from context. We still override this decision
and always unlink when a temp file is used.

Signed-off-by: Dave Reisner <dreisner@archlinux.org>
Signed-off-by: Dan McGee <dan@archlinux.org>
2011-08-19 09:43:19 -05:00
Dan McGee c885a953eb Enhance and utilize database status flags
* Move is_local standalone field to status enum
* Create VALID/INVALID flag pair
* Create EXISTS/MISSING flag pair

With these additional fields, we can be more intelligent with database
loading and messages to the user. We now only warn once if a sync
database does not exist and do not continue to try to load it once we
have marked it as missing.

The reason for the flags existing in pairs is so the unknown case can be
represented. There should never be a time when both flags in the same
group are true, but if they are both false, it represents the unknown
case. Care is taken to always manipulate both flags at the same time.

Signed-off-by: Dan McGee <dan@archlinux.org>
2011-08-18 10:47:41 -05:00
Dan McGee 11f4a7a48e Only check necessary signatures and checksums
The precedence goes as follows: signature > sha256sum > md5sum

Add some logic and helper methods to check what we have available when
loading a package, and then only check what is necessary to verify the
package. This should speed up sync database verifies as we no longer
will be doing both a checksum and a signature validation.

Signed-off-by: Dan McGee <dan@archlinux.org>
2011-08-15 15:15:11 -05:00
Dan McGee a628feee46 Parse conflicts/provides/replaces at database load time
We did this with depends way back in commit c244cfecf6 in 2007. We
can do it with these fields as well.

Of note is the inclusion of provides even though only '=' is supported-
we'll parse other things, but no guarantees are given as to behavior,
which is more or less similar to before since we only looked for the
equals sign.

Also of note is the non-inclusion of optdepends; this will likely be
resolved down the road.

The biggest benefactors of this change will be the resolving code that
formerly had to parse and reparse several of these fields; it only
happens once now at load time. This does lead to the disadvantage that
we will now always be parsing this information up front even if we never
need it in the split form, but as these are uncommon fields and our
parser is quite efficient it shouldn't be a big concern.

Signed-off-by: Dan McGee <dan@archlinux.org>
2011-08-15 12:56:41 -05:00
Dan McGee ebb2e36cc4 Load and allow access to sha256sum
This adds a field in the package struct for this checksum type as well
as allowing access via the API to it. The frontend is now able to
display any read value. Note that this does not implement any use or
verification of the value internally.

Signed-off-by: Dan McGee <dan@archlinux.org>
2011-08-15 12:11:55 -05:00
Dan McGee cbaff216b3 Don't trim whitespace when reading database entries
We don't write with extra or unknown whitespace, so there is little
reason for us to trim it when reading either. This also fixes the
hopefully never encountered "paths that start or end with spaces" issue,
for which two pactests have been added. The tests also contain other
evil characters that we have encountered before and handle just fine,
but it doesn't hurt to ensure we don't break such support in the future.

Signed-off-by: Dan McGee <dan@archlinux.org>
2011-08-02 08:20:34 -04:00
Dan McGee 70d6fe6632 Clean up my debug logger mess
Signed-off-by: Dan McGee <dan@archlinux.org>
2011-07-19 00:23:42 -05:00
Dan McGee 05f7c0280e Fix test suite when GPGME is disabled
As noted by Allan, we failed pretty hard if gpgme was compiled out. With
these changes, only sign001.py fails. This can/will be fixed later once
we beef up the test suite with more signing tests anyway.

Signed-off-by: Dan McGee <dan@archlinux.org>
2011-07-18 21:06:24 -05:00
Dave Reisner 3eec745910 absorb some _alpm_download params into payload struct
Restore some sanity to the number of arguments passed to _alpm_download
and curl_download_internal.

Signed-off-by: Dave Reisner <dreisner@archlinux.org>
2011-07-05 23:00:02 -04:00
Dave Reisner 6dc71926f9 lib/dload: prevent large file attacks
This means creating a new struct which can pass more descriptive data
from the back end sync functions to the downloader. In particular, we're
interested in the download size read from the sync DB. When the remote
server reports a size larger than this (via a content-length header),
abort the transfer.

In cases where the size is unknown, we set a hard upper limit of:

* 25MiB for a sync DB
* 16KiB for a signature

For reference, 25MiB is more than twice the size of all of the current
binary repos (with files) combined, and 16KiB is a truly gargantuan
signature.

Signed-off-by: Dave Reisner <dreisner@archlinux.org>
2011-07-05 22:58:55 -04:00
Dave Reisner 6c9b82e72a dload: handle irregular URLs
URLs might end with a slash and follow redirects, or could be a
generated by a script such as /getpkg.php?id=12345. In both cases, we
may have a better filename that we can write to, taken from either
content-disposition header, or the effective URL.

Specific to the first case, we write to a temporary file of the format
'alpmtmp.XXXXXX', where XXXXXX is randomized by mkstemp(3). Since this
is a randomly generated file, we cannot support resuming and the file is
unlinked in the event of an interrupt.

We also run into the possibility of changing out the filename from under
alpm on a -U operation, so callers of _alpm_download can optionally pass
a pointer to a *char to be filled in by curl_download_internal with the
actual filename we wrote to. Any sync operation will pass a NULL pointer
here, as we rely on specific names for packages from a mirror.

Fixes FS#22645.

Signed-off-by: Dave Reisner <d@falconindy.com>
2011-07-05 22:58:27 -04:00
Dan McGee c748eadc80 Allow invalid sync DBs to be returned by the library
They are placeholders, but important for things like trying to re-sync a
database missing a signature. By using the alpm_db_validity() method at
the right time, a client can take the appropriate action with these
invalid databases as necessary.

In pacman's case, we disallow just about anything that involves looking
at a sync database outside of an '-Sy' operation (although we do check
the validity immediately after). A few operations are still permitted-
'-Q' ops that don't touch sync databases as well as '-R'.

Signed-off-by: Dan McGee <dan@archlinux.org>
2011-07-05 10:13:20 -05:00
Dan McGee 7af0ab1cde signing: move to new signing verification and return scheme
This gives us more granularity than the former Never/Optional/Always
trifecta. The frontend still uses these values temporarily but that will
be changed in a future patch.

* Use 'siglevel' consistenly in method names, 'level' as variable name
* The level becomes an enum bitmask value for flexibility
* Signature check methods now return a array of status codes rather than
  a simple integer success/failure value. This allows callers to
  determine whether things such as an unknown signature are valid.
* Specific signature error codes mostly disappear in favor of the above
  returned status code; pm_errno is now set only to PKG_INVALID_SIG or
  DB_INVALID_SIG as appropriate.

Signed-off-by: Dan McGee <dan@archlinux.org>
2011-07-05 10:13:20 -05:00
Dan McGee 1ce7f39ad7 Merge remote-tracking branch 'allan/ALPM'
Conflicts:
	lib/libalpm/be_local.c
	lib/libalpm/be_package.c
	lib/libalpm/conflict.c
	lib/libalpm/diskspace.c
	lib/libalpm/dload.c
	lib/libalpm/remove.c
2011-07-03 14:44:57 -05:00
Dan McGee de8b9a85a5 be_sync: make READ_NEXT() a no-arg macro
We passed in 'line', but not 'buf.line'. In addition, the macros
building off of READ_NEXT() assume variable names anyway. Since we only
use these macros in one function, might as well simplify them.

Signed-off-by: Dan McGee <dan@archlinux.org>
2011-07-03 13:07:19 -05:00
Allan McRae afc96f2ab3 Prefix _alpm_errno_t members with ALPM
Signed-off-by: Allan McRae <allan@archlinux.org>
2011-07-02 02:01:39 +10:00
Allan McRae ca43fdd92f Prefix alpm_loglevel_t members with ALPM
Signed-off-by: Allan McRae <allan@archlinux.org>
2011-07-02 02:01:38 +10:00
Dan McGee 23a2d2c16a Make alpm_db_get_sigverify_level() public
Signed-off-by: Dan McGee <dan@archlinux.org>
2011-06-30 16:16:41 -05:00
Dan McGee 68284da0d7 Add an alpm_db_get_valid() public function
This allows one to check if a database is valid or invalid.

Signed-off-by: Dan McGee <dan@archlinux.org>
2011-06-30 16:05:20 -05:00
Allan McRae 8a04bc25a1 Rename pmpkg_t to alpm_pkg_t
Signed-off-by: Allan McRae <allan@archlinux.org>
2011-06-28 23:26:39 +10:00
Allan McRae 939d5a9511 Rename pmdb_t to alpm_db_t
Signed-off-by: Allan McRae <allan@archlinux.org>
2011-06-28 14:16:12 +10:00
Allan McRae 64c1cf7921 Rename pmhandle_t to alpm_handle_t
Signed-off-by: Allan McRae <allan@archlinux.org>
2011-06-28 14:04:00 +10:00
Dan McGee 777bdc6c50 Make sync DB reading a bit more flexible
We can reorganize things a bit to not require reading a directory-only
entry first (or at all). This was noticed while working on some pactest
improvements, but should be a good step forward anyway.

Also make _alpm_splitname() a bit more generic in where it stores the
data it parses.

Signed-off-by: Dan McGee <dan@archlinux.org>
2011-06-24 14:04:39 -05:00
Dan McGee 4f8ae2bab6 Don't require a transaction for sync DB updates
Instead, just do the required locking directly in the backend in calls
to alpm_db_update().

Signed-off-by: Dan McGee <dan@archlinux.org>
2011-06-24 04:11:38 -05:00
Dan McGee 79e98316ea Add a 'valid' flag to the database object
Start by converting all of our flags to a 'status' bitmask (pkgcache
status, grpcache status). Add a new 'valid' flag as well. This will let
us keep track if the database itself has been marked valid in whatever
fashion.

For local databases at the moment we ensure there are no depends files;
for sync databases we ensure the PGP signature is valid if
required/requested. The loading of the pkgcache is prohibited if the
database is invalid.

Signed-off-by: Dan McGee <dan@archlinux.org>
2011-06-24 03:46:12 -05:00
Dan McGee 1150d9e15a Move database 'version' check to registration time
This is another step toward doing both local database validation
(ensuring we don't have depends files) and sync database validation (via
signatures if present) when the database is registered.

Signed-off-by: Dan McGee <dan@archlinux.org>
2011-06-24 03:31:32 -05:00
Dan McGee db3b86e7f3 Do database signature checking at load time
This is the ideal place to do it as all clients should be checking the
return value and ensuring there are no errors. This is similar to
pkg_load().

We also add an additional step of validation after we download a new
database; a subsequent '-y' operation can potentially invalidate the
original check at registration time.

Note that this implementation is still a bit naive; if a signature is
invalid it is currently impossible to refresh and re-download the file
without manually deleting it first. Similarly, if one downloads a
database and the check fails, the database object is still there and can
be used. These shortcomings will be addressed in a future commit.

Signed-off-by: Dan McGee <dan@archlinux.org>
2011-06-24 02:34:50 -05:00
Dan McGee d589a7b5db Prevent segfault when parsing unexpected sync database file
This doesn't fix the real (bigger) problem of failing to parse sync
databases without directory entries, but it does prevent the parser from
segfaulting when the first desc file encountered did not have a
directory entry, among other conditions.

Signed-off-by: Dan McGee <dan@archlinux.org>
2011-06-24 01:36:47 -05:00
Kerrick Staley d980bd403d Documented purpose of be_*.c
Added a line to the top of each of be_local.c, be_package.c, and
be_sync.c indicating their purposes.

Signed-off-by: Kerrick Staley <mail@kerrickstaley.com>
Signed-off-by: Dan McGee <dan@archlinux.org>
2011-06-20 00:17:54 -05:00
Dan McGee ee015f086f Ensure handle is valid and pm_errno is reset when calling into API
We didn't do due diligence before and ensure prior pm_errno values
weren't influencing what happened in further ALPM calls. I observed one
case of early setup code setting pm_errno to PM_ERR_WRONG_ARGS and that
flag persisting the entire time we were calling library code.

Add a new CHECK_HANDLE() macro that does two things: 1) ensures the
handle variable passed to it is non-NULL and 2) clears any existing
pm_errno flag set on the handle. This macro can replace many places we
used the ASSERT(handle != NULL, ...) pattern before.

Several other other places only need a simple 'set to zero' of the
pm_errno field.

Signed-off-by: Dan McGee <dan@archlinux.org>
2011-06-14 10:01:08 -05:00
Dan McGee f6700d5c98 alpm_db_update(): refactor out sync dir create/check
This was a lot of stuff that can stand by itself for the most part.

Signed-off-by: Dan McGee <dan@archlinux.org>
2011-06-13 19:41:37 -05:00
Dan McGee 52bffd2457 Switch all logging to use handle directly
This is the last user of our global handle object. Once again the diff
is large but the functional changes are not.

Signed-off-by: Dan McGee <dan@archlinux.org>
2011-06-13 19:41:16 -05:00
Dan McGee e2aa952689 Move pm_errno onto the handle
This involves some serious changes and a very messy diff, unfortunately.

Signed-off-by: Dan McGee <dan@archlinux.org>
2011-06-13 19:38:38 -05:00
Dan McGee 17a6ac5675 Require handle argument to all alpm_option_(get|set)_*() methods
This requires a lot of line changes, but not many functional changes as
more often than not our handle variable is already available in some
fashion.

Signed-off-by: Dan McGee <dan@archlinux.org>
2011-06-09 14:24:45 -05:00
Dan McGee 2668782db2 Plug a memory leak
Introduced by me in commit cc25576f8b.

Signed-off-by: Dan McGee <dan@archlinux.org>
2011-06-08 02:48:43 -05:00
Dan McGee cc25576f8b Use standard errno codes in return from _alpm_archive_fgets
This allows us to not require the context (e.g. handle) when calling
this function. Also beef up the checks in the two callers of this
function to bail if the last return code is not ARCHIVE_EOF, which is
the expected value.

This requires a change to one of the pactest return codes and the
overall result of the test, but results in a much safer operating
condition whereby invalid database entries will stop the operation.

Signed-off-by: Dan McGee <dan@archlinux.org>
2011-06-03 14:37:51 -05:00
Dan McGee 307a6de17a Remove global handle from some package and db code
Signed-off-by: Dan McGee <dan@archlinux.org>
2011-06-03 12:44:01 -05:00
Dan McGee de36c5fac4 Push down extern handle variable to files that need it
This will make the patching process less invasive as we start to remove
this variable from all source files.

Signed-off-by: Dan McGee <dan@archlinux.org>
2011-06-03 12:23:32 -05:00
Dan McGee 7f98460e37 Add handle attribute to pmpkg_t struct
Similar to what we just did for the database; this will make it easy to
always know what handle a given package originated from.

Signed-off-by: Dan McGee <dan@archlinux.org>
2011-06-03 12:13:22 -05:00
Dan McGee c47d25d74b Add handle attribute to pmdb_t struct
This is the first step in a long process to remove our dependence on the
global handle variable we currently share in libalpm, with the goal to
make things a bit more thread-safe and re-entrant.

Signed-off-by: Dan McGee <dan@archlinux.org>
2011-06-03 12:06:25 -05:00
Dan McGee 78cbc045c1 Remove ALPM_LOG_FUNC macro
The usefulness of this is rather limited due to it not being compiled
into production builds. When you do choose to see the output, it is
often overwhelming and not helpful. The best bet is to use a debugger
and/or well-placed fprintf() statements.

Signed-off-by: Dan McGee <dan@archlinux.org>
2011-06-03 11:48:24 -05:00
Dan McGee 2cd79bc853 Remove sync DB reregister check
It's your own damn fault if you do this, and this code is remnants from
an old time when we weren't very good at coding.

Signed-off-by: Dan McGee <dan@archlinux.org>
2011-05-05 15:26:19 -05:00
Dan McGee 6b308d89f9 Merge branch 'maint' 2011-05-05 13:25:10 -05:00
Dan McGee b14c5477e5 Ensure populate error return codes are consistent
It must be -1 to differentiate it from a number of packages loaded
count.

Signed-off-by: Dan McGee <dan@archlinux.org>
2011-05-05 12:10:51 -05:00
Dan McGee 3045f09ef4 Logging changes during DB load
The switch from FUNCTION to DEBUG was ill-advised inside the local
database load. Instead, add a DEBUG level logger to both local and sync
database loads that shows the number of packages processed.

Signed-off-by: Dan McGee <dan@archlinux.org>
2011-05-05 12:08:59 -05:00
Allan McRae 7680f46157 Deal with unused function parameters correctly
This started off removing the "(void)foo" hacks to work around
unused function parameters and ended up fixing every warning
generated by -Wunused-parameter.

Dan: rename to UNUSED.

Signed-off-by: Allan McRae <allan@archlinux.org>
Signed-off-by: Dan McGee <dan@archlinux.org>
2011-05-04 15:53:49 -05:00
Dan McGee 225acbbff1 Rein in the complexity of the signature type
Given that we offer no transparency into the pmpgpsig_t type, we don't
really need to expose it outside of the library, and at this point, we
don't need it at all. Don't decode anything except when checking
signatures. For packages/files not from a sync database, we now just
read the signature file directly anyway.

Also push the decoding logic down further into the check method so we
don't need this hanging out in a less than ideal place. This will make
it easier to conditionally compile things down the road.

Signed-off-by: Dan McGee <dan@archlinux.org>
2011-04-24 10:48:33 -05:00
Dan McGee 9579879b1b libalpm/dload: major refactor of signature downloading
There's a lot of related moving parts here:
* Iteration through mirrors is moved back to the calling functions. This
  allows removal of _alpm_download_single_file and _alpm_download_files.
* The download function gets a few more arguments to influence behavior.
  This allows several different scenarios to customize behavior:
  - database
  - database signature (req'd and optional)
  - package
  - package via direct URL
  - package signature via direct URL (req'd and optional)
* For databases, we need signatures from the same mirror, so structure
  the code accordingly.

Some-inspiration-from: Dave Reisner <d@falconindy.com>
Signed-off-by: Dan McGee <dan@archlinux.org>
2011-04-22 17:08:33 -05:00
Dan McGee 1ff04b980f be_sync: use _alpm_db_get_sigverify_level()
Signed-off-by: Dan McGee <dan@archlinux.org>
2011-04-22 15:58:09 -05:00
Dan McGee 442e1420f9 Rename gpgsig struct fields for clarity
Signed-off-by: Dan McGee <dan@archlinux.org>
2011-04-20 20:11:26 -05:00
Dan McGee 791928dc48 Header inclusion cleanup
This does touch a lot of things, and hopefully doesn't break things on
other platforms, but allows us to also clean up a bunch of crud that no
longer needs to be there.

Signed-off-by: Dan McGee <dan@archlinux.org>
2011-04-20 20:09:13 -05:00
Dan McGee c5addd94e3 Merge branch 'maint'
Conflicts:
	lib/libalpm/be_sync.c
	lib/libalpm/db.c
	src/pacman/util.c
2011-04-05 00:49:30 -05:00
Rémy Oudompheng c377107346 Fix compatibility with older versions of libarchive.
There is no reason to not support versions of libarchive that lack
ARCHIVE_COMPRESSION_UU. Distributions should work properly without
this.

Signed-off-by: Rémy Oudompheng <remy@archlinux.org>
Signed-off-by: Dan McGee <dan@archlinux.org>
2011-04-02 12:39:04 -05:00
Dan McGee 39fd8bc318 Ensure dbpath is not null when populating sync database
We didn't do this sanity check before trying to open an archive. If
the alpm dbpath wasn't set, the sync database dbpath would be NULL,
causing us to hang indefinitely in archive_read_open_filename() rather
than erroring out.

We already have a corresponding check in local_db_populate().

The following program will test this case, and hangs before this patch
without the call to set_dbpath:

	int main(int argc, char *argv[]) {
		alpm_initialize();
		// alpm_option_set_dbpath("/var/lib/pacman/");
		pmdb_t *core = alpm_db_register_sync("core");
		pmpkg_t *pkg = alpm_db_get_pkg(core, "pacman");
		return 0;
	}

Signed-off-by: Dan McGee <dan@archlinux.org>
2011-04-01 14:31:50 -05:00
Dan McGee 20c4928ee1 Ignore upcoming new values in sync backend
PGPSIG and SHA256SUM are new and we can safely ignore them for now if
we come across them.

Signed-off-by: Dan McGee <dan@archlinux.org>
2011-04-01 12:31:48 -05:00
Allan McRae a8b22e16ef Do not reuse old signature
After updating a database, remove the old signature to prevent it
being used in validation if the new signature fails to download.

Signed-off-by: Allan McRae <allan@archlinux.org>
2011-03-23 22:58:32 -05:00
Allan McRae 38da050f31 Download and verify package database signatures
If signature verification is needed, attempt to download a signature
file for a repo when it is updated. Return an error if unable to
download signature only when checking is mandatory, or if signature is
invalid.

TODO: At the moment the database signature is only checked on download.
Should we do anything with a database if it fails to be verified to prevent
its future usage?

Signed-off-by: Allan McRae <allan@archlinux.org>
2011-03-23 22:58:00 -05:00
Dan McGee 39ce9b3afc Actually read PGPSIG field in sync DB code
Signed-off-by: Dan McGee <dan@archlinux.org>
2011-03-23 03:07:08 -05:00
Dan McGee a31d091fb3 repo-add: add sha256sum values to repo database
Implements FS#23103. Also modify libalpm so it ignores this value
without any warning as we know it is likely to exist.

Signed-off-by: Dan McGee <dan@archlinux.org>
2011-03-23 02:18:48 -05:00
Dan McGee 0303b26b1e Style change: return(x) --> return x
This was discussed and more or less agreed upon on the mailing list. A
huge checkin, but if we just do it and let people adjust the pain will
end soon enough. Rebasing should be relatively straighforward for anyone
that sees conflicts; just be sure you use the new return style if
possible.

The following semantic patch was used to do the change, along with some
hand-massaging in order to preserve parenthesis where appropriate:

The semantic match that finds this problem is as follows, although some
hand-massaging was done in order to keep parenthesis where appropriate:
(http://coccinelle.lip6.fr/)

// <smpl>
@@
expression a;
@@
- return(a);
+ return a;

// </smpl>

A macros_file was also provided with the following content:

Additional steps taken, mainly for ASSERT() macros:
$ sed -i -e 's#return(NULL)#return NULL#' lib/libalpm/*.c
$ sed -i -e 's#return(-1)#return -1#' lib/libalpm/*.c

Signed-off-by: Dan McGee <dan@archlinux.org>
2011-03-20 19:49:45 -05:00
Allan McRae f6716ae94a Use sane umask for repo db downloads
Fixes FS#23343.

Signed-off-by: Allan McRae <allan@archlinux.org>
Signed-off-by: Dan McGee <dan@archlinux.org>
2011-03-20 11:17:00 -05:00
tuxce b48f718417 Correctly parse %DELTAS% entries in sync DB
We erroniously dropped the call to _alpm_delta_parse() when macro-izing,
causing segfaults for repos that provide deltas. Addresses FS#23314.

Signed-off-by: Dan McGee <dan@archlinux.org>
2011-03-17 10:51:44 -05:00
Allan McRae f2023176f6 Do not print warning with files entry in sync db
repo-add can add a "files" entry into the sync db.  Currently we
do nothing with this file, so explicitly skip it to prevent
unknown database file warnings.

Signed-off-by: Allan McRae <allan@archlinux.org>
Signed-off-by: Dan McGee <dan@archlinux.org>
2011-03-07 13:54:38 -06:00
Dan McGee 09ce8b446c Fix some easy to find double translations
A lot of these were places that should have used the same message but
didn't, or were very easy to convert to using the same message and
letting some of the burden off of the translators.

Signed-off-by: Dan McGee <dan@archlinux.org>
2011-02-28 17:50:23 -06:00
Dan McGee f45369800a Check local DB version before continuing transaction
Ensure we have a local DB version that is up to par with what we expect
before we go down any road that might modify it. This should prevent
stupid mistakes with the 3.5.X upgrade and people not running
pacman-db-upgrade after the transaction as they will need to.

Signed-off-by: Dan McGee <dan@archlinux.org>
2011-02-28 10:46:00 -06:00
Dan McGee d1cc1ef6c3 Fix some database size estimation problems
* Use stat() and not lstat(); we don't care for the size of the symlink if
  it is one, we want the size of the reference file.
* FS#22896, fix local database estimation on platforms that don't abide by
  the nlink assumption for number of children.
* Fix a missing newline on an error message.

Signed-off-by: Dan McGee <dan@archlinux.org>
2011-02-15 16:58:08 -06:00
Pang Yan Han be9a60a338 Handle null pkgcache for local/sync db_populate()
In sync_db_populate() and local_db_populate(), a NULL db->pkgcache is not
caught, allowing the functions to continue instead of exiting.

A later alpm_list_msort() call which uses alpm_list_nth() will thus traverse
invalid pointers in a non-existent db->pkgcache->list.

pm_errno is set to PM_ERR_MEMORY as _alpm_pkghash_create() will only return
NULL when we run out of memory / exceed max hash table size. The local/sync
db_populate() functions are also exited.

Signed-off-by: Pang Yan Han <pangyanhan@gmail.com>
Signed-off-by: Dan McGee <dan@archlinux.org>
2011-02-07 20:21:34 -06:00
Dan McGee 021085624e Change default sync hash table sizing to 66% full
Since the sync database never changes size once we initialize it, we
allow it to be filled a bit more. This reduces the overall memory
footprint needed by the hash table.

Signed-off-by: Dan McGee <dan@archlinux.org>
2011-02-04 09:55:45 +10:00
Allan McRae f8fdce6cb0 Read pkgcache into hash
Read the package information for sync/local databases into a pmpkghash_t
structure.

Provide a alpm_db_get_pkgcache_list() method that returns the list from
the hash object.  Most usages of alpm_db_get_pkgcache are converted to
this at this stage for ease of implementation.  Review whether these are
better accessing the hash table directly at a later stage.

Signed-off-by: Allan McRae <allan@archlinux.org>
2011-02-04 09:55:45 +10:00
Dan McGee 5dae577a87 Get estimated package count when populating databases
This works for both local and sync databases in slightly different ways. For
the local database, we can use the directory hard link count on the local/
folder. For sync databases, we use the archive size coupled with some
computed average per-package sizes to determine an estimate.

This is currently a dead assignment once calculated, but could be used to
set the initial size of a hash table.

Signed-off-by: Dan McGee <dan@archlinux.org>
Signed-off-by: Allan McRae <allan@archlinux.org>
2011-02-04 09:55:45 +10:00
Dan McGee 87240dae6d Fix locale.h/setlocale inclusion with --disable-nls
Noted in FS#22697. When I factored out _alpm_parsedate() into a common
function, I didn't move the <locale.h> include properly, causing a build
failure when NLS is disabled and this header isn't automatically included
everywhere.

Signed-off-by: Dan McGee <dan@archlinux.org>
2011-02-01 09:59:12 -06:00
Xavier Chantry ed1aef7bc5 libalpm: fix db_update documentation
return codes were mixed up

Signed-off-by: Dan McGee <dan@archlinux.org>
2011-01-31 09:36:30 -06:00
Dan McGee fe76c353af Fix memory leak and error code in DB reading
We were returning a package error code rather than a DB one, and we
would leak the archive memory if the database file didn't exist.

Signed-off-by: Dan McGee <dan@archlinux.org>
2011-01-24 21:46:42 -06:00
Dan McGee bf46e04614 Remove epoch as an independent field
Instead, go the same route we have always taken with version-release in
libalpm and treat it all as one piece of information. Makepkg is the only
script that knows about epoch as a distinct value; from there on out we will
parse out the components as necessary.

This makes the code a lot simpler as far as epoch handling goes. The
downside here is that we are tossing some compatibility to the wind;
packages using force will have to be rebuilt with an incremented epoch to
keep their special status.

Signed-off-by: Dan McGee <dan@archlinux.org>
2011-01-21 19:30:45 -06:00
Dan McGee 3863e48788 Add a likely_pkg hint argument to sync_db_read
In most (all?) cases, we will process all files for a given sync database
entry sequentially. The code currently does an _alpm_pkg_find() for every
file in the database, but we had the "current" package readily available.
Shift some local variables around a bit to expose this to sync_db_read() and
use it if the package is the correct one.

On my system, this cuts calls to _alpm_pkg_find() from 20,769 to 10,349
calls during a -Qu operation, and results in a ~30% speedup of the same
operation (0.35 sec -> 0.27 sec). This benefit should be apparent anywhere
we read in the full contents of the sync databases.

Signed-off-by: Dan McGee <dan@archlinux.org>
2011-01-21 10:11:59 -06:00
Allan McRae f966f3a834 Use size_t for alpm_list sizes
There is a lot of swtiching between size_t and int for alpm_list sizes
in the codebase.   Start converting these to all be size_t by adjusting
the return type of alpm_list_count and fixing all additional warnings
given by -Wconversion that are generated by this change.

Dan: a few more small changes to ensure things compile, adjusting some
printf format string characters to accommodate the larger size on x86_64.

Signed-off-by: Allan McRae <allan@archlinux.org>
Signed-off-by: Dan McGee <dan@archlinux.org>
2011-01-07 21:15:46 -06:00
Dan McGee 4bc6ed56aa Refactor old date parsing into single method
We've managed to duplicate this four times at this point, so make it a
method in util.c instead.

Signed-off-by: Dan McGee <dan@archlinux.org>
2011-01-07 20:55:05 -06:00
Allan McRae d288240426 Update copyright years for 2011
Signed-off-by: Allan McRae <allan@archlinux.org>
Signed-off-by: Dan McGee <dan@archlinux.org>
2011-01-07 18:47:37 -06:00
Allan McRae 81dd9d3ebc Detect undefined PATH_MAX
POSIX does not require PATH_MAX be defined when there is not actual
limit to its value.  This affects HURD based systems.  Work around
this by defining PATH_MAX to 4096 (as on Linux) when this is not
defined.

Also, clean up inclusions of limits.h and remove autoconf check for
this header as we do not use macro shields for its inclusion anyway.

Signed-off-by: Allan McRae <allan@archlinux.org>
Signed-off-by: Dan McGee <dan@archlinux.org>
2010-12-29 19:24:13 -06:00
Dan McGee f2dff08600 Overhaul archive fgets function
The old function was written in a time before we relied on it for nearly
every operation. Since then, we have switched to the archive backend and now
fast parsing is a big deal.

The former function made a per-character call to the libarchive
archive_read_data() function, which resulted in some 21 million calls in a
typical "load all sync dbs" operation. If we instead do some buffering of
our own and read the blocks directly, and then find our newlines from there,
we can cut out the multiple layers of overhead and go from archive to parsed
data much quicker.

Both users of the former function are switched over to the new signature,
made easier by the macros now in place in the sync backend parsing code.

Performance: for a `pacman -Su` (no upgrades available),
_alpm_archive_fgets() goes from being 29% of the total time to 12% The time
spent on the libarchive function being called dropped from 24% to 6%.

This pushes _alpm_pkg_find back to the title of slowest low-level function.

Signed-off-by: Dan McGee <dan@archlinux.org>
2010-12-21 14:58:17 -06:00
Dan McGee 8ac7f7e6e5 Use macros in sync DB parsing
This simplifies a lot of the repetative code and makes it obvious where the
tricky or different ones are (e.g. depends, dates). It also makes it
significantly easier to change the way this code works in the future.

There should be no functional change with this patch.

Signed-off-by: Dan McGee <dan@archlinux.org>
2010-12-20 18:02:47 -06:00
Dan McGee c00e05992e Remove non-public functions from header
And rename accordingly.

Signed-off-by: Dan McGee <dan@archlinux.org>
2010-12-13 21:16:47 -06:00
Dan McGee 94d3d665f0 Mark sync_db_read() as static
Signed-off-by: Dan McGee <dan@archlinux.org>
2010-12-13 21:14:03 -06:00
Dan McGee 1ff8118212 Create sync/ DB directory if it does not exist
Rather than error out, this is easy enough. Looks quite similar to the code
in be_local for creating the local directory.

Signed-off-by: Dan McGee <dan@archlinux.org>
2010-12-12 22:41:02 -06:00
Dan McGee 3030542d10 Make reading from any file possible in sync DB
Whether it be "desc", "depends", or "deltas", it really doesn't matter-
treat them all the same and have the ability to read any data from any file
in that list. This continues the work in a44c7b8956.

Signed-off-by: Dan McGee <dan@archlinux.org>
2010-12-12 20:38:20 -06:00