building OpenBSD packages in bulk
There are quite a few steps necessary to build packages on a cluster. They are:
- Choose master machine setup and create partitions.
- Setup chrooted builds on the master.
- Add slaves and do a full bulk.
- Clean up and do subsequent bulks.
- Perform additional maintenance.
Setup a master machine with enough room for a chroot, say
. Assuming you are using a cluster of
machines, this chroot should contain NFS exportable partitions for distfiles,
plists, and packages (one single partition can be used for simplicity). A full
setup requires on the order of 50GB for distfiles and 50GB for packages.
It is possible to build packages without a chroot, but the space requirement
difference is negligible (a full OpenBSD
less than 1GB), and having everything chrooted means you may install useful
tools to help with the process outside of the chroot (for instance
Reserve one "scratch" partition under the chroot for WRKOBJDIR (for
instance, mfs, async, or SSD). This partition should be roughly 10GB if you
want to be able to build all ports. This can often double as
under the chroot.
Alternately, you can setup your whole chroot as a scratch partition, and reserve
one more permanent space under it for distfiles, packages, and plists.
Choose a strategy for the ports tree itself. There must be a copy under
. You can either copy it from outside
the chroot, have it in an NFS partition, or manually make sure all machines on
the cluster have the same ports tree (cvs checkout, rsync ...).
Note that logs are only produced on the master, and thus do not need to be nfs
exportable, nor even inside the chroot.
now comes with default users for package builds,
namely _pbuild and _pfetch.
The default login.conf(5)
appropriate for most setups, but _pbuild's datasize-cur will need to be bumped
for a few ports, like pypy. Likewise, maxproc-cur is too small for machines
with more than 4-6 cpus.
Note that _pbuild does not need network access, and is now blocked by default in
systems do not need any kind of
setup for bulk ports
builds, as dpb(1)
is run as root
and drops permissions appropriately.
However, you may still want to setup
for root, if you want to
manually fix ports, as PORTS_PRIVSEP
Populate the initial chroot with
. Point DISTDIR,
PACKAGE_REPOSITORY, PLIST_REPOSITORY, WRKOBJDIR to appropriate locations.
Pay attention to nodev and wxallowed warnings. A chroot setup that can't have
devices won't work at all. A setup without wxallowed in /usr/local and
WRKOBJDIR won't build a lot of things.
Check that this setup can build ports by running
as root. Fix any issues related to file ownership at this point.
If your WRKOBJDIR is a temporary partition, make sure it belongs to
_pbuild:_pbuild after a reboot.
Create identical slave machines with the same release material. Have them import
the NFS partitions from the master, they don't need root access to the
partitions. Set up ssh(1)
the master can connect to the slaves, using ssh protocol 2, as root,
preferably without a password or passphrase (however,
uses a master connection, so
a password would be required just once per host).
Note that code on slave machines will only run as _pbuild (during builds) or
root (during dependency installation). Slave machines only require highly
restricted network access. They just need to act as nfs clients to the master
and to be reachable through ssh from the master.
Use a similar proot(1)
populate each slave.
You should now be able to build ports on the slaves. A simple config will just
Check that the full config can still build ports.
You're now ready for a full bulk. Beware that even fast configs (3 amd64 with 8
cores each) may take over 24 hours to finish. It's generally appropriate to
Before running the next bulk, you should set up rotating logs and move the
PACKAGE_REPOSITORY away. Save the PLIST_REPOSITORY and DISTDIR though.
PLIST_REPOSITORY will catch problems in packing-lists.
used to store sha256(1)
necessary to reorder files inside packages to speed updates up.
The DISTDIR contains history information as well as DISTDIR/build-stats to speed
further runs up.
How you wipe things out depends on your setup. If you run
again each time, most
things will get cleaned up automatically
). Note that known
directories such as WRKOBJDIR do not get cleaned up automatically, so you may
want to set up a STARTUP_SCRIPT in
should be run occasionally since the DISTDIR will continue growing.
be run occasionally to find out conflicts and dependency issues.