| DPB(1) | General Commands Manual | DPB(1) |
dpb — distributed
ports builder
dpb |
[-acemqRrsUuvx] [-A
arch] [-B
chroot] [-b
logfile] [-C
pathlist] [-D
PARAM=value]
[-F m]
[-f m]
[-h hosts]
[-I pathlist]
[-J p]
[-j n]
[-L logdir]
[-l lockdir]
[-M threshold]
[-P pathlist]
[-p parallel]
[-S logfile]
[-X pathlist]
[pathlist ...] |
dpb is used to build ports on a cluster of
machines, or on a single machine with several cores.
dpb walks the ports tree to figure out dependencies,
and starts building ports as soon as it can.
dpb will run with sensible defaults if
used without options. Note, however, that it will produce logs, lock files,
packages, and package installations.
If run as non-root, dpb will warn. The
preferred way is to run it as root (and preferably under a chroot).
dpb will then change its identity to different users
as needed. See ‘THE SECURITY MODEL OF DPB’ for details.
dpb can be restricted to a subset of the
tree by giving it pathlist ... to build as
parameters.
A pathlist is either a
pkgpath(7) to build, or a filename that
contains pkgpaths (one per line). pathlist parameters
can also take the form filename*scale in order to
multiply the weights of all pkgpath(7)
in a file by a given scale, or
pkgpath=value, in order to set the weight of a given
pkgpath(7) to a specific value.
dpb supports ‘hot-fixes’: if
a particular port errors out, it is possible to fix the problem, remove the
corresponding lockfile, and dpb will pick it up
without needing to be stopped and restarted.
In order to build on a cluster, the ports tree itself should be identical on each machine (shared through NFS or copied at start).
Some directories must be shared:
PACKAGE_REPOSITORY, DISTDIR,
and PLIST_REPOSITORY. The
WRKOBJDIR and LOCKDIR should
be local to each machine, and on a high-speed partition.
Also note that dpb's logs and locks are
managed by the main dpb process, which runs locally,
and hence those directories do not need to be shared on the cluster.
Some log files ("rolling logs") are kept from one run to the run and stored under ${DISTDIR}/build-stats.
Option -h file is
used to specify hosts to use, where file may contain
lots of information, but can be as simple as a list of hosts to use, one
host per line (however, it is recommended to also include a
STARTUP script).
Most filenames will go through some control sequence expansions. For instance, the default logdir location can be specified as %p/logs/%a. The following sequences are recognized:
%a%ddpb, GMtime, formatted as
yyyy-mm-dd@hh:mm:ss.%f%hdpb.%L%p%tdpb.%$dpb process .Options are as follows:
-A
arch-a-B
chroot-b
logfile-C
pathlist-c-D
PARAM=valuedpb should clean work directories
even if the port errored out.dpb is not run as root.dpb create a unix socket of the given name
for external control. Defaults to ‘%L/control-%h-%$’. If
no socket is wanted, explicitly set CONTROL to
empty.dpb will use the
BUILD_ONCE optimization (see
bsd.port.mk(5)) if run with
-a: pseudo-flavors that disable subpackages
and are not necessary for bootstrap will be disabled, so that the same
port is built once, as far as possible. This flag disables that
optimization, which might be desirable if you want to build a small
subset of packages which would pull in the kitchen sink
otherwise.dpb will clean old locks from dpb
running on the same host that no longer exist, provided they didn't
end in error. This is usually the right thing to do after a crash, or
after killing dpb abruptly. Sometimes, one may want manual control
over which locks to remove.dpb to
figure out old distfiles and update
%f/history.-e.DISTFILES (default for
dpb -f). If 1, will
also fetch extra SUPDISTFILES (default for
dpb -F).dpb is run a second time after a problem
during the first run.dpb output and
replay it later at a faster rate. Defaults to
%L/term-report.log, can be set to nothing to
disable.dpb call
syslog(3) on every task start/end
while creating packages. This does produce lots of messages, it is
intended to route the logging on another machine, while tracking down
panics and other hangs.-s.-e-F
mBROKEN and
ONLY_FOR_ARCHS information. Create
m localhost jobs for fetching files.-f
mdpb memory footprint by a
lot.-h
hosts
espie@aeryn jobs=4 arch=i386
Lines starting with a known variable name such as
STARTUP=path
FETCH_JOBS=5
The special hostname DEFAULT can be used to preset defaults. It should be used at the start of the file.
Use localhost to specify the local
machine. dpb will special-case it and not use
ssh(1) to connect.
Properties are as follows:
-J
option.USE_MFS=‘Yes’, assuming the
ports tree has been configured so that
WRKOBJDIR_MFS points to a memory filesystem.
thr is the sum, in KBytes, of ports that will be
allowed to build in memory. dpb understands
suffixes, such as -M 2G
or -M 500M.
Note that you should always allow for some margin, as
dpb makes its decision based on the size
information collected during previous builds, so in cases of
significant updates, the work directory size will usually grow.
-q (no checksum) option.-p
option.dpb won't
bother calling fine-grained steps for patch/configure/fake. It will go
straight to build and package instead. Defaults to 120 seconds.There are no fine-grained options to control ssh(1) options, as those can be specified through virtual host declarations in ssh_config(5).
-I
pathlist-J
p-aXI options to delete automatically installed
packages that are currently not needed.
dpb keeps track of list of
dependencies on a given host, by storing each dependency list in the
lockfile corresponding to the package being built.
To avoid a race condition between the
depends and junk stages,
dpb allows only one job on a given host to be in
the depends ... junk stages
at one time, by using a per-host lock.
Defaults to 150. Can be disabled by setting to 0.
Some ports, most notably cmake-based, have an annoying dependency handling bug: they compute their makefile dependencies based on all include files present, not just the ones that are actually enabled. Those ports' build may be broken by a junk phase that removes some unused includes that were added as makefile prerequisites. Those ports should be annotated with DPB_PROPERTIES = nojunk until that bug is fixed: while a port with the ‘nojunk’ property is building, junk will be postponed.
Those ports will be marked with a ‘!’ in the display, to make it more obvious why junk seems to be ineffective.
Note that the ‘nojunk’ property is still active for ports in error, in the belief that trivial fixes can be made that will allow the port build to finish.
-j
n-L
logdir-l
lockdir-M
thresholdWRKOBJDIR_MFS (see
bsd.port.mk(5)).
threshold is the sum, in KBytes, of ports allowed to
build there.-m-P
pathlist-p
parallelRun big jobs on several cores on the same host, by using MAKE_JOBS=k.
Once such a job has started, dpb will
not start new jobs on the same host until the big job has stolen enough
cores from other finishing jobs.
Only big ports which are safe for parallel building (annotated with DPB_PROPERTIES = parallel in their Makefile) will be affected.
It is advisable to set k to an integral fraction of the number
of cores available on a given host. parameter can
be an integer, or of the form ‘/n’, in which case,
dpb will set k to a fraction of the total number
of jobs on the machine, but never below 2.
Defaults to ‘/2’.
-q-RNote that -R won't always work, as
rebuilding a package when another version is already installed is not
supported. Building in a chroot is strongly recommended.
-r-S
logfile-sdpb will actually not always compute
new sizes for known directories, but mostly for new ones, or when the
package name changes.-U-u-X
pathlist-xdpb figures out in which order to build
things on the fly, and constantly displays information relative to what's
currently building. There's a list of what is currently running, one line
per job. Those jobs are ordered in strict chronological order, which means
that long running builds will tend to percolate to the top of the list.
Normal jobs look like this:
www/mozilla-firefox(build) [9452] 41% unchanged for 92 seconds
This contains:
And fetch jobs look like this:
<dist-3.0.tgz(#1) [4321] 25%
This contains:
MASTER_SITE being triedThis is followed by a host line, containing the name of each host used by dpb. Host names may be tagged with kde3 or kde4. They are followed by a ‘`-'’ for unresponsive hosts, and the pid of the ssh master for distant hosts.
This ends with a summary display:
-f is
used.If those three lists are empty, they won't even show up. Packages in errors may be followed by a ‘!’ if they prevent junk from happening.
Note that those numbers refer to pkgpaths known to
dpb. In general, those numbers will be slightly
higher than the actual number of packages being built, since several paths
may lead to the same package.
dpb uses some heuristics to try to
maximise the queue as soon as possible. There are also provisions for a
feedback-directed build, where information from previous builds can be used
to try to build long-running jobs first.
Similarly, fetches will use the continue option of ftp(1), since distfiles are checksummed after the fetch anyways.
However, on machines with lots of cores, the basic scheduling
heuristics yields a tail of very small jobs, where
dpb will mostly wait on
pkg_add(1) to solve dependencies.
Starting with OpenBSD 5.5, a new mechanism
(squiggles) was introduced to counter-balance this effect: big machines
devote some of their cores to ‘squiggles’, jobs that walk the
queue in reverse, thus building smallest ports first. As a result, small
ports are built as a trickle alongside the largest ports, thus offsetting
the negative effect of the exponential queue for a large part.
Note that ‘squiggles’ can be a non-integral value, usually lower than 1, in which case they represent the fraction of cores that should be affected to squiggles, as decided randomly at the start of each build. 0.7 or 0.8 might be a good choice for dual core machines.
The bsd.port.mk(5)
variable DPB_PROPERTIES may hold several annotations
that only dpb will look at. These properties are as
follows:
dpb from scheduling anything else
on the same host after it starts building.MAKE_JOBS
and several build slots.MAKE_JOBS and lots of build slots.When dpb is run as root, it uses a
privilege drop model instead of the dangerous privilege elevation model of
doas(1). When run as root, by default,
_pbuild is used as the build and log user, and
_pfetch is used as the fetch user.
dpb as root.dpb will drop privileges for every operation
except pkg_add(1),
pkg_delete(1) and the
STARTUP script.chroot -u build_user
/build_root (with /build_root =
/ if there is no actual chroot needed). It must
have read access to ${DISTDIR} and ${PORTSDIR}, and write access to
${WRKOBJDIR}, ${PACKAGE_REPOSITORY}, and ${PLIST_REPOSITORY}. It does not
require network access.dpb creates local directories as root, then gives
them to the appropriate user.dpb still uses the normal ports tree
mechanism while building, which includes LOCKDIR.
When starting up dpb will normally detect stale
locks from old dpb runs, and remove them. If this does not happen, builds
will stay stuck in their initial stage, that is:
show-prepare-results, patch,
build depending on the port. A telltale message
‘Awaiting lock ...’ can be found in the corresponding logfile
paths/pkgpath.log
In addition, when building a package, dpb
produces a lockfile in the locks directory, whose name is deduced from the
basic pkgpath with slashes replaced by dots. This lockfile is filled with
such info as the build start time or the host, or the needed dependencies
for this pkgpath.
The lockfile will also contain the name of a parent pkgpath, for paths that were discovered as dependencies. This is particularly useful for bogus paths, where it would be hard to know where the path came from otherwise.
At the end of a successful build, these lockfiles are removed. The lock will stay around in case of errors. (raw value from wait(2)), and the name of the next task in the build pipeline (with todo=<nothing> in case of failure during clean-up). Normal list of tasks is: depends prepare fetch patch configure build fake package clean.
At the end of each job, dpb rechecks the
locks directory for existing lockfiles. If some locks have vanished, it will
put the corresponding paths back in the queue and attempt another build.
This eases manual repairs: if a package does not build, the user
can look at the log, go to the port directory, fix the problem, and then
remove the lock. dpb will pick up the ball and keep
building without interruption.
It is perfectly safe to run several dpb in
parallel on the same machine. This is not optimal, since each
dpb ignores the others, and only uses the lock info
to avoid the other's current work, but it can be handy: in an emergency, one
can start a second dpb to obtain a specific package
right now, in parallel with the original dpb.
Note that dpb is very careful not to run
two builds from the same pkgpath at the same time, even on different
machines: in some cases, MULTI_PACKAGES and FLAVOR combinations may lead to
the same package being built simultaneously, and since the package
repository is shared, this can easily lead to trouble.
Handling of shared log files and history is also done very carefully by systematically appending to files or using atomic mv operations.
For obvious reasons, this won't work as well with masters running on distinct machines sharing their logs through NFS.
There are some various interdependencies in package builds that can be hard to trace in case something goes wrong. Refer to summary.log to fix those specific issues.
dpb now maintains a list of
pkgpath-per-host that are currently building in the
affinity directory of its log directory, along with
building-in-memory status.
That information is only wiped out when a given build finishes successfully.
Otherwise dpb will try to restart that
build on the same host, which can be handy if you interrupt
dpb while it is building a large port, or if you
remove a lock after fixing a problem.
Currently, kde3 and kde4 can't be built simultaneously. Conflicting ports have been annotated with DPB_PROPERTIES=tag:kde3 , DPB_PROPERTIES=tag:kde4 respectively.
dpb now keeps track of those tags, and
will postpone ports with the wrong tag while a given host is used by the
other tag.
This heavily relies on the junk stage to clean-up hosts periodically, and it can even forcibly provoke a junk stage even if junk=0.
This ‘force-junk’ stage is actually implemented as a pseudo path called junk-proxy, which only does junk.
In order for builds to proceed gracefully, machines should start in a clean slate, without kde3 or kde4 installed.
As a special-case, failing ports with a kde3 or kde4 tag will not interfere with clean-up, so that hosts do not get locked down to a specific tag. This also means that their dependencies may vanish before human intervention addresses the problem.
This is supposed to be a temporary hack, as kde4 is large and having official packages helps a great deal in debugging it.
By default (see CONTROL),
dpb will create a Unix socket at
%L/control-%h-%$, only accessible by
LOG_USER, that can accept a few commands, e.g., usable
as nc -U path
Currents commands are as follows:
addhost
hostlineaddpath
fullpkgpath ...byedontclean
pkgpath ...helpstatsstatus
fullpkgpath ...wipe
fullpkgpath ...wipehost
hostname ...dpb, including running jobs, locks, and affinity
information.dpb periodically checks for a file named
stop in its log directory. If this file exists, then
it won't start new jobs, and shutdown when the current jobs are finished
unless -q.
dpb also checks for files named
stop-<hostname> in its log directory. If such
a file exists, then it won't start new jobs on the corresponding
machine.
Apart from producing packages, dpb may
create temporary files as
${FULLDISTDIR}/${DISTFILE}.part.
In fetch mode (-f and
-F), dpb populates
${DISTDIR}/by_cipher/sha256 with links. It also uses
${DISTDIR}/distinfo and
${DISTDIR}/history as a ‘permanent
log’:
dpb -a), the fetch
engine knows precisely which files are needed by the build (and their
checksums). Anything that is
ts SHA256 (file) = value
with ts a timestamp from Unix epoch.
When cleaning up old files, with a tool such as clean-old-distfiles(1), it is vital to check both the checksum and the file name: since mirroring stores permanent links under by_cipher, files which are still needed will appear in history under their old checksums, as an indication the link should be removed, but possibly not the file itself.
If ${DISTDIR} ever becomes corrupted,
removing ${DISTDIR}/distinfo will force
dpb into checking all files again.
All those files belong to the FETCH_USER if it is defined. They should be readable for the build_user.
dpb also records rolling build statistics
under ${DISTDIR}/build-stats/${ARCH}, and uses them
automatically in the absence of -b
logfile. That file belongs to the
LOG_USER if it is defined.
If -s is used, size information for
successful builds will be recorded under
${DISTDIR}/build-stats/${ARCH}-size (by default,
location adjustable with -S
sizelog). This is then reused for the mfs threshold
option. That file also belongs to the LOG_USER if it
is defined.
dpb also maintains a list of pkgpath
frequencies
${DISTDIR}/build-stats/${ARCH}-dependencies, filled
at end of LISTING if -a. This list will be
automatically reused when restarting a build: a quick LISTING of the most
important dependencies will happen before the general LISTING, in order to
prime further LISTING steps with most common ports first.
dpb will also create a large number of log
files under ${PORTSDIR}/logs/${ARCH}, which will
belong to LOG_USER if it is defined:
dpb reads existing affinity
information, and records it in that log, together with its pid. This log
just exists to verify, along with engine.log,
whether correct affinity was heeded.The detailed timing info gives a run-down of the build, with clean, fetch, prepare, patch (actually extract+patch), configure, build, fake, package, clean detailed timing info. Note that the actual build time starts at ‘extract’ and finishes at ‘package’.
dpb, plus a timestamp
of the log entry.
Please note that the engine is no longer run after each package build event because of performance considerations, so the ‘Q’ and ‘I’ changes may be delayed by a few ‘B’.
-J counts the number of dependencies
directly added to decide when to run pkg_delete
-a. This file sums up how many ports were built,
and how many ports had dependencies each time dpb
decides to junk.dpb are computationally intensive,
such as the engine runs to determine new stuff that can be built, and the
actual display reports.
Both those activities are rate-limited, so that
dpb doesn't run its engine at each new package
build, and doesn't update its display every time there is a phase
change.
Lines tagged with ‘ENG’ correspond to the engine; lines tagged with ‘REP’ correspond to the display reports.
Lines ending with a dash ‘-’ correspond to new activity that didn't trigger a computation.
Other lines will feature a plus ‘+’ for normal runs, or an exclamation point ‘’! for forced runs, followed by two numbers: the next timestamp at which we'll be allowed to run, and a measure of how much time it took to run this pass.
That information is mostly relevant while
dpb is building lots of small packages very
quickly.
-s.dpb creating new jobs.In that last case, summary.log contains a chain of dependencies leading to the problematic package, or in case of build cycles, stopping at the first loop.
dpb is setting up
hosts, getting essential data from the ports tree, running a
STARTUP script, collecting base library
signatures.dpb detects a
"frozen" port has happened outside of
dpb's purview, namely because the ports tree
itself has that specific port locked without dpb's
knowledge. See bsd.port.mk(5),
portlock(1).dpb process, which needs to
know exactly which dependencies are used at a given point, so that
junk can clean up the host correctly. In particular,
junk will not clean up dependencies already
scheduled for installation. Ports that do not obtain the lock on first try
are put to sleep.dpb maintains dependencies for a given host
globally, it coalesces depends lists together.dpb still
keeps track of attempted junks.dpb can't start a new build job.dpb can't start a new fetch job.dpb performs best with lots of paths to
build. When just used to build a few ports, there's a high risk of
starvation as there are bottlenecks in parts of the tree.
Fetch jobs don't deal with checksum changes yet: if a fetch fails
because of a wrong checksum, if you update the distinfo file and remove the
lock, dpb won't pick it up.
Note that dpb does not manage installed
packages in any intelligent way, it will just call
pkg_add(1) during its depend stage to
install its dependencies. With -u, it will call
pkg_add -r. With -U, it will call pkg_add -r -D
installed, but there is nothing else going on. This is especially true when
using -R, ensure the machine is clean of possibly
older packages first, or run dpb with
-U.
In particular -R and
-J together may lead to strange issues.
On heterogeneous networks, calibration of build info and choice of speed factors is not perfect, and somewhat a dark art. Using distinct speed factors on a build log that comes from a single machine works fine, but using the build info coming from several machines does not work all that well.
dpb should check
/usr/include and
/usr/X11R6/include for consistency, but it
doesn't.
When a host fails consistency check, there is not yet a way to
re-add it after fixing the problem. You have to stop
dpb, cleanup and restart.
The default limits in login.conf are too small for bulk builds on any kind of parallel machines. Bump number of processes, file descriptors, and memory.
Even though dpb tries really hard to check
heterogeneous networks for sanity (checking shared libraries and .la files),
it is still dependent on the user to make sure all the hosts build ports the
same way.
Make sure your NFS setup is consistent. The ports dir itself should be exported or synchronized. Distfiles, the package repository, and the plist repository should be exported, but WRKOBJDIR should not be on NFS unless you have absolutely no choice, or if you exhibit deep masochistic tendencies. Pay particular attention to discrepancies in /etc/mk.conf.
Also, dpb connects to external hosts
through ssh(1), relying on
ssh_config(5) for any special
cases.
When fetching distfiles, dpb may freeze
and spin in a tight loop while the last distfiles are being fetched. This is
definitely a bug, which has been around for quite some time, which is a bit
difficult to reproduce, and hasn't been fixed yet. So if
dpb stops updating its display right around the end
of fetch, you've hit the bug. Just kill dpb and
restart it.
The original dpb command was written by
Nikolay Sturm. This version is a complete rewrite from scratch using all the
stuff we learnt over the years to make it better.
Marc Espie <espie@openbsd.org>
| July 1, 2019 | OpenBSD-current |