Using FreeBSD’s Tinderbox as a package builder

Tinderbox setup

The machine I’m using is currently being used to test port updates. It has a bunch of jails for the -STABLE branches and a copy of the ports tree that I make changes to when testing port updates. I decided to use this machine for my package builder but this meant keeping things separated. So for package building I have the following set up in Tinderbox:

• A jail for 8.2-RELEASE (RELENG_8_2).
• A jail for 9.0-RELEASE (RELENG_9_0), when that gets branched.
• A separate ports tree that I can keep pristine and update automatically without affecting my other ports work.

I won’t document how to do that. The Tinderbox README covers it in plenty of detail.

Index generation

If you’re just doing a pristine ports tree, with no OPTIONS or other environment tweaks, and you don’t care that the INDEX file may be slightly newer than your package set, you don’t need to do this. I have some OPTIONS set and I wanted the INDEX to exactly match the versions of the available packages, so I’m building my own INDEX file.

I checked the Tinderbox archives for the best way to do this. Others seem to be doing it using a hook on the ports tree update. The problem with this is that you need to do some extra work to make sure any OPTIONS or environment changes are included, and if you’re doing it for multiple OS versions you’ll need to cover that too (otherwise it’ll build the INDEX according to your Tinderbox host’s OS version).

The solution I came up with was to make a small custom port. It builds INDEX and installs it to /usr/local. I build this inside each build I’m using for my package building and the result is a package containing the INDEX file that fits my criteria (OPTIONS, environment, and matches my ports tree exactly).

Here’s the port’s Makefile. The symlink line is only needed because Puppet, which I use, looks for INDEX.bz2 rather than INDEX-8.bz2.

PORTNAME=       makeindex
PORTVERSION=    0
CATEGORIES=     ports-mgmt
MASTER_SITES=   # none
DISTFILES=      # none

MAINTAINER=     tdb@FreeBSD.org
COMMENT=        Generate INDEX file

USE_PERL5=      yes # make index requires perl

PLIST_FILES=    ${INDEXFILE}.bz2 INDEX.bz2

do-build:
        cd ${PORTSDIR} && make index INDEXDIR=${WRKDIR} -DINDEX_PRISTINE
        bzip2 -9 ${WRKDIR}/${INDEXFILE}

do-install:
        ${INSTALL_DATA} ${WRKDIR}/${INDEXFILE}.bz2 ${PREFIX}
        ln -s ${INDEXFILE}.bz2 ${PREFIX}/INDEX.bz2

.include <bsd.port.mk>

Package builds

The next step is to tie the INDEX generation together with updating the ports tree and building packages. It’s a pretty simple process; update the ports tree, generate and install the new INDEX file and then build any new packages. Below is the script I use to do this, and here are a few useful notes:

• $TB/tdb/autobuildports is a list of ports that I want to build, one per line, in the format “category/portname”.
• $TB/tdb/makeindex is the port discussed in the previous section.
• I use the -norebuild flag to tinderbuild to ensure I don’t rebuild the leaf ports unless necessary.
• The last step after the for loop is mostly so I can check what it’s done, and isn’t necessary for things to work.

#!/bin/sh

TB=/u1/tinderbox
PT=FreeBSD_auto
portlist=$TB/tdb/autobuildports

PATH=/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin:$PATH export PATH

$TB/scripts/tc updatePortsTree -p $PT

for b in `ls $TB/builds | grep $PT`; do
        rsync -rlpvc --delete --force --exclude=CVS/ \
                $TB/tdb/makeindex/. \
                $TB/portstrees/$PT/ports/ports-mgmt/makeindex
        $TB/scripts/tc addPort \
                -b $b -d ports-mgmt/makeindex
        $TB/scripts/tc tinderbuild \
                -b $b -nullfs ports-mgmt/makeindex

        cd $TB/packages/$b && tar -zxvf All/makeindex-0.tbz INDEX\*

        for p in `cat $portlist`; do
                echo "===> $p on $b"
                $TB/scripts/tc addPort \
                        -b $b -d $p
                $TB/scripts/tc tinderbuild \
                        -b $b \
                        -nullfs -norebuild \
                        $p
        done

        cd $TB/packages/$b/All && ls > $TB/packages/$b/All.new
        echo "New packages:"
        comm -1 -3 $TB/packages/$b/All.last $TB/packages/$b/All.new
        mv $TB/packages/$b/All.new $TB/packages/$b/All.last
done

I run this script on a daily basis from cron.

Portmaster setup

The final step is installing these packages. I could do this by hand using pkg_add, but I prefer to use Portmaster. It’ll handle upgrades too. I use the following config in my portmaster.rc file which sets all the useful options for working with binary packages.

Portmaster will pull the INDEX file automatically as required. I picked /var/db/portmaster as the temporary area to put packages in, but you could use another place if /var is space limited.

# Do not create temporary backup packages before pkg_delete (-B)
NO_BACKUP=Bopt

# Only install packages (-PP or --packages-only)
PM_PACKAGES=only

# Use the INDEX file instead of /usr/ports (--index-only)
PM_INDEX=pm_index
PM_INDEX_ONLY=pm_index_only

# Delete packages after they are installed (--delete-packages)
PM_DELETE_PACKAGES=pm_delete_packages

# Local paths
PACKAGES=/var/db/portmaster
INDEXDIR=/var/db/portmaster
MASTER_SITE_INDEX=http://my.tinderbox.server/packages/8.2-RELEASE-FreeBSD/
PACKAGESITE=http://my.tinderbox.server/packages/8.2-RELEASE-FreeBSD/

So that’s it. I can now run portmaster category/port to install a new port or portmaster -a to upgrade everything and I’ll get the latest packages built using my custom options.

My final point is that this is all still a little fresh. I only just wrote it and I haven’t been using it long. So there’s undoubtedly something I’ve missed. You’ve been warned!