format for OpenBSD binary
Binary packages for OpenBSD can be created using pkg_create(1) and are usually manipulated using pkg_add(1), pkg_mklocatedb(1), or pkg_info(1).
The basic underlying format is an archive following the ustar specification that can be handled with tar(1) and compressed using gzip(1).
Package names always end in “.tgz”; the file name itself should conform to packages-specs(7).
Note that the base distribution tarballs of OpenBSD (e.g. baseXX.tgz, compXX.tgz, ...) are not binary packages fit for pkg_add(1).
All types of archive contents can be present in a package, including files, directories, hardlinks, symlinks, fifos, block and character devices.
In order to allow just-in-time extraction, packages always begin with a table of contents, named +CONTENTS. This table of contents can be read using the API described in OpenBSD::PackingList(3p).
All the remaining information in the archive should be referenced in the packing-list, including all relevant information: symlinks destinations, special permissions, and file owners (pkg_create(1) and pkg_add(1) actually enforce this). See pkg_create(1) for annotation details.
This table of contents is always followed by a few special files, some of which are optional: the package description (+DESC), a display message (+DISPLAY), etc.
The basic ustar format has some limitations with respect to file names. Packages now use the "extended record specification" (header type x) for long links and long file names. Other extended ustar headers are currently recognized, but not supported.
Starting with OpenBSD 5.5, the compressed archive may be composed of several gzip(1) archives concatenated together. gzip(1) doesn't mind, and tar(1) is happy as long as the uncompressed stream is sane. This allows for faster signing and better rsync properties.
Starting with OpenBSD 5.6, tarballs are stored "out-of-order": each archive entry will match an entry in the packing-list (and all file-like entries will be matched), but the order will be adjusted so that most recently changed files come first, in order to allow faster updates.
Starting with OpenBSD 5.7, by default,
timestamps are stored directly in the packing-list as
@ts annotations. The files in the archive will have
a null timestamp.
The combination of the LRU archive order, null timestamps and compression in fixed chunks starting from the end, means a substantial amount of data doesn't change from snapshot to snapshot, thus reducing the pressure on mirroring bandwidth.
PACKING LIST ANNOTATIONS
User annotations are described in pkg_create(1). The following annotations are automatically inserted during package creation and installation:
- List of architectures for which this package is intended. This corresponds
-Aarches of pkg_create(1)
@comment pkgpath=path ftp=yes/no
- Historical accident. This specific comment encodes the actual
-DFTP arguments to pkg_create(1).
- Record a dependency declared using the option
- Record a digital signature of the packing-list, synthetized by pkg_add(1) from signify(1) output.
- Added after a file entry by
packageto record that the entry is actually a hard link.
- Used internally to record the settings of
- Set the name of the package. This name is potentially different than the name of the file it came in, and is used when keeping track of the package for later deinstallation. pkg_create(1) will derive this field from the package file name.
- Some options are automatically inserted by the package tools:
- Set by fw_update(8) to trigger firmware-specific handling. In particular, firmware is hidden from normal updates.
- Record that a package has been explicitly installed by the user, and
not as a result of a dependency look-up. Refer to
-aoption for details.
- Added after a file entry by pkg_create(1) to record the files's cryptographic checksum, as a sha256 digest encoded in base64.
- Internal annotation necessary to identify packages signed with signify(1) keys, as those keys don't carry any identity.
- Added after a file entry by pkg_create(1) to record a file size.
- Added after a file entry by pkg_create(1) to record that the entry is actually a symbolic link.
- Original location of the package, automatically recorded in installed packages by pkg_add(1).
- Added after a file entry to record the actual file timestamp, instead of storing it as tarball meta-info. The package tools read and process that annotation correctly, whereas tar(1) will only see files dated from the epoch.
- Record a global (system) version number for the package. This is built
-Voptions from pkg_create(1), and triggers updates when it changes.
- Record a library requirement declared using the option
Note that most of these annotations cannot be added manually, as pkg_create(1) will error out.
All information within a package is checksummed, using SHA256 since OpenBSD 4.4. During creation and installation, meta-information, such as file owners and permissions, are also checked: any important stuff that isn't recorded in the packing-list is an error.
Packing-lists can be signed. If a signature is found, then it will be checked during installation, and failure to verify will prevent the package from installing correctly.
Starting with OpenBSD 6.1,
gzip(1) header signatures are the only supported format. This allows
for ‘just-in-time’ signature checking, as the binary data is
checked in 64K bytes long chunks.
pkg_add(1), pkg_create(1), pkg_info(1), pkg_sign(1), packages(7), packages-specs(7)
Packages are valid gzip'ed ustar archives that can be extracted using tar(1). In particular, hardlink names should be valid, and all items will extract to different names. However, it may be a bit difficult to make sense of the package contents without peeking at the packing-list.