{"id":417,"date":"2011-10-25T11:53:07","date_gmt":"2011-10-25T11:53:07","guid":{"rendered":"https:\/\/www.bishnet.net\/tim\/blog\/?p=417"},"modified":"2011-11-02T15:21:00","modified_gmt":"2011-11-02T15:21:00","slug":"using-freebsds-tinderbox-as-a-package-builder","status":"publish","type":"post","link":"https:\/\/www.bishnet.net\/tim\/blog\/2011\/10\/25\/using-freebsds-tinderbox-as-a-package-builder\/","title":{"rendered":"Using FreeBSD&#8217;s Tinderbox as a package builder"},"content":{"rendered":"<h3>Tinderbox setup<\/h3>\n<p>The machine I&#8217;m using is currently being used to test port updates. It has a bunch of jails for the -STABLE branches and a copy of the ports tree that I make changes to when testing port updates. I decided to use this machine for my package builder but this meant keeping things separated. So for package building I have the following set up in <a href=\"http:\/\/tinderbox.marcuscom.com\/\">Tinderbox<\/a>:<\/p>\n<ul>\n<li>A jail for 8.2-RELEASE (RELENG_8_2).<\/li>\n<li>A jail for 9.0-RELEASE (RELENG_9_0), when that gets branched.<\/li>\n<li>A separate ports tree that I can keep pristine and update automatically without affecting my other ports work.<\/li>\n<\/ul>\n<p>I won&#8217;t document how to do that. The <a href=\"http:\/\/tinderbox.marcuscom.com\/README\/index.html\">Tinderbox README<\/a> covers it in plenty of detail.<\/p>\n<h3>Index generation<\/h3>\n<p>If you&#8217;re just doing a pristine ports tree, with no OPTIONS or other environment tweaks, and you don&#8217;t care that the <code>INDEX<\/code> file may be slightly newer than your package set, you don&#8217;t need to do this. I have some OPTIONS set and I wanted the <code>INDEX<\/code> to exactly match the versions of the available packages, so I&#8217;m building my own <code>INDEX<\/code> file.<\/p>\n<p>I checked the Tinderbox archives for the best way to do this. Others seem to be doing it using a hook on the ports tree update. The problem with this is that you need to do some extra work to make sure any OPTIONS or environment changes are included, and if you&#8217;re doing it for multiple OS versions you&#8217;ll need to cover that too (otherwise it&#8217;ll build the <code>INDEX<\/code> according to your Tinderbox host&#8217;s OS version).<\/p>\n<p>The solution I came up with was to make a small custom port. It builds <code>INDEX<\/code> and installs it to \/usr\/local. I build this inside each build I&#8217;m using for my package building and the result is a package containing the <code>INDEX<\/code> file that fits my criteria (OPTIONS, environment, and matches my ports tree exactly).<\/p>\n<p>Here&#8217;s the port&#8217;s Makefile. The symlink line is only needed because <a href=\"http:\/\/puppetlabs.com\/\">Puppet<\/a>, which I use, looks for INDEX.bz2 rather than INDEX-8.bz2.<\/p>\n<blockquote>\n<pre>PORTNAME=       makeindex\r\nPORTVERSION=    0\r\nCATEGORIES=     ports-mgmt\r\nMASTER_SITES=   # none\r\nDISTFILES=      # none\r\n\r\nMAINTAINER=     tdb@FreeBSD.org\r\nCOMMENT=        Generate INDEX file\r\n\r\nUSE_PERL5=      yes # make index requires perl\r\n\r\nPLIST_FILES=    ${INDEXFILE}.bz2 INDEX.bz2\r\n\r\ndo-build:\r\n        cd ${PORTSDIR} &amp;&amp; make index INDEXDIR=${WRKDIR} -DINDEX_PRISTINE\r\n        bzip2 -9 ${WRKDIR}\/${INDEXFILE}\r\n\r\ndo-install:\r\n        ${INSTALL_DATA} ${WRKDIR}\/${INDEXFILE}.bz2 ${PREFIX}\r\n        ln -s ${INDEXFILE}.bz2 ${PREFIX}\/INDEX.bz2\r\n\r\n.include &lt;bsd.port.mk&gt;<\/pre>\n<\/blockquote>\n<h3>Package builds<\/h3>\n<p>The next step is to tie the <code>INDEX<\/code> generation together with updating the ports tree and building packages. It&#8217;s a pretty simple process; update the ports tree, generate and install the new <code>INDEX<\/code> file and then build any new packages. Below is the script I use to do this, and here are a few useful notes:<\/p>\n<ul>\n<li><code>$TB\/tdb\/autobuildports<\/code> is a list of ports that I want to build, one per line, in the format &#8220;category\/portname&#8221;.<\/li>\n<li><code>$TB\/tdb\/makeindex<\/code> is the port discussed in the previous section.<\/li>\n<li>I use the <code>-norebuild<\/code> flag to <code>tinderbuild<\/code> to ensure I don&#8217;t rebuild the leaf ports unless necessary.<\/li>\n<li>The last step after the <code>for<\/code> loop is mostly so I can check what it&#8217;s done, and isn&#8217;t necessary for things to work.<\/li>\n<\/ul>\n<blockquote>\n<pre>#!\/bin\/sh\r\n\r\nTB=\/u1\/tinderbox\r\nPT=FreeBSD_auto\r\nportlist=$TB\/tdb\/autobuildports\r\n\r\nPATH=\/bin:\/sbin:\/usr\/bin:\/usr\/sbin:\/usr\/local\/bin:\/usr\/local\/sbin:$PATH export PATH\r\n\r\n$TB\/scripts\/tc updatePortsTree -p $PT\r\n\r\nfor b in `ls $TB\/builds | grep $PT`; do\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 rsync -rlpvc --delete --force --exclude=CVS\/ \\\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 $TB\/tdb\/makeindex\/. \\\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 $TB\/portstrees\/$PT\/ports\/ports-mgmt\/makeindex\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 $TB\/scripts\/tc addPort \\\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 -b $b -d ports-mgmt\/makeindex\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 $TB\/scripts\/tc tinderbuild \\\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 -b $b -nullfs ports-mgmt\/makeindex\r\n\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 cd $TB\/packages\/$b &amp;&amp; tar -zxvf All\/makeindex-0.tbz INDEX\\*\r\n\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 for p in `cat $portlist`; do\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 echo \"===&gt; $p on $b\"\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 $TB\/scripts\/tc addPort \\\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 -b $b -d $p\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 $TB\/scripts\/tc tinderbuild \\\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 -b $b \\\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 -nullfs -norebuild \\\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 $p\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 done\r\n\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 cd $TB\/packages\/$b\/All &amp;&amp; ls &gt; $TB\/packages\/$b\/All.new\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 echo \"New packages:\"\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 comm -1 -3 $TB\/packages\/$b\/All.last $TB\/packages\/$b\/All.new\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 mv $TB\/packages\/$b\/All.new $TB\/packages\/$b\/All.last\r\ndone<\/pre>\n<\/blockquote>\n<p>I run this script on a daily basis from cron.<\/p>\n<h3>Portmaster setup<\/h3>\n<p>The final step is installing these packages. I could do this by hand using <code>pkg_add<\/code>, but I prefer to use Portmaster. It&#8217;ll handle upgrades too. I use the following config in my <code>portmaster.rc<\/code> file which sets all the useful options for working with binary packages.<\/p>\n<p>Portmaster will pull the <code>INDEX<\/code> file automatically as required. I picked <code>\/var\/db\/portmaster<\/code> as the temporary area to put packages in, but you could use another place if <code>\/var<\/code> is space limited.<\/p>\n<blockquote>\n<pre># Do not create temporary backup packages before pkg_delete (-B)\r\nNO_BACKUP=Bopt\r\n\r\n# Only install packages (-PP or --packages-only)\r\nPM_PACKAGES=only\r\n\r\n# Use the INDEX file instead of \/usr\/ports (--index-only)\r\nPM_INDEX=pm_index\r\nPM_INDEX_ONLY=pm_index_only\r\n\r\n# Delete packages after they are installed (--delete-packages)\r\nPM_DELETE_PACKAGES=pm_delete_packages\r\n\r\n# Local paths\r\nPACKAGES=\/var\/db\/portmaster\r\nINDEXDIR=\/var\/db\/portmaster\r\nMASTER_SITE_INDEX=http:\/\/my.tinderbox.server\/packages\/8.2-RELEASE-FreeBSD\/\r\nPACKAGESITE=http:\/\/my.tinderbox.server\/packages\/8.2-RELEASE-FreeBSD\/<\/pre>\n<\/blockquote>\n<p>So that&#8217;s it. I can now run <code>portmaster category\/port<\/code> to install a new port or <code>portmaster -a<\/code> to upgrade everything and I&#8217;ll get the latest packages built using my custom options.<\/p>\n<p>My final point is that this is all still a little fresh. I only just wrote it and I haven&#8217;t been using it long. So there&#8217;s undoubtedly something I&#8217;ve missed. You&#8217;ve been warned!<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Tinderbox setup The machine I&#8217;m using is currently being used to test port updates. It has a bunch of jails for the -STABLE branches and a copy of the ports tree that I make changes to when testing port updates. I decided to use this machine for my package builder but this meant keeping things separated. So for package building &hellip;<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[3,4],"tags":[140,132,131],"class_list":["post-417","post","type-post","status-publish","format-standard","hentry","category-computing","category-freebsd","tag-freebsd","tag-package-builder","tag-tinderbox"],"aioseo_notices":[],"_links":{"self":[{"href":"https:\/\/www.bishnet.net\/tim\/blog\/wp-json\/wp\/v2\/posts\/417","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.bishnet.net\/tim\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.bishnet.net\/tim\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.bishnet.net\/tim\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.bishnet.net\/tim\/blog\/wp-json\/wp\/v2\/comments?post=417"}],"version-history":[{"count":18,"href":"https:\/\/www.bishnet.net\/tim\/blog\/wp-json\/wp\/v2\/posts\/417\/revisions"}],"predecessor-version":[{"id":435,"href":"https:\/\/www.bishnet.net\/tim\/blog\/wp-json\/wp\/v2\/posts\/417\/revisions\/435"}],"wp:attachment":[{"href":"https:\/\/www.bishnet.net\/tim\/blog\/wp-json\/wp\/v2\/media?parent=417"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.bishnet.net\/tim\/blog\/wp-json\/wp\/v2\/categories?post=417"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.bishnet.net\/tim\/blog\/wp-json\/wp\/v2\/tags?post=417"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}