Date: 06 April 2005
Updating a FreeBSD box is fairly simple but it can still be a pain when you have lots of them to upgrade. I have put together this little tutorial as an “It works for me” example. There may be better ways of doing this.
The first thing you will need is a building machine. This box will download and build the kernel, userland apps and libraries for every machine. Once built, we’ll share them via NFS to the other boxes.
You will need to have installed the system source before hand. If you haven’t, you can use sysinstall to install the source package. You all also need cvsup to fetch the updated source. (It’s in the ports collections as net/cvsup and net/cvsup-without-gui.) The appendix A of the handbook has instructions on configuring cvsup.
cd /usr/src make update
Once the update has finished you can start building. We’ll start by building the userland.
Building the Kernel
Let’s take a minute before we build the kernel and talk about how we deal with handling different kernels for different boxes. At some point in your career you’ll decide that you need a custom kernel for one or more of your servers. This is fine. The handbook suggests using a custom kernel configuration for each box. That will work but it becomes impractical if you have lots of boxes. In most cases, the kernel configs will be the same, or at least, very similar. If possible, try to factor out those common things into a standard kernel that you can use on many boxes. For example, I have a standard kernel for most of my boxes, a SMP kernel for my multi-proc boxes and a kernel for routers.
You select the kernel you want on my setting KERNCONF in /etc/make.conf.
The builder machine needs to know which kernels to build so we’ll put a list instead of just one like above. builder’s kernel is the first one on the list. Note: Don’t forget to build a GENERIC kernel.
KERNCONF=STANDARD STANDARD-SMP ROUTER GENERIC
Once you’re kernels are configured, it’s time to build.
Now that you’ve built your new version of FreeBSD, it’s time to get them to your other boxes. The simple way is via NFS. Security Note: NFS has several security issues that are beyond the scope of this article. Make sure you take appropriate steps to secure your NFS exports.
You’ll need to export /usr/src and /usr/obj to the remote machines. Add /usr to /etc/exports.
/usr -maproot=root -alldirs servers
Note: You can export /usr read-only for added security.
Now that the building machine is done we can look at the client setup.
Let’s add /usr/src and /usr/obj to /etc/fstab. This is not strictly required but can make life a little easier.
builder:/usr/src /usr/src nfs rw 0 0 builder:/usr/obj /usr/obj nfs rw 0 0
Now we can mount /usr/src with a simple mount /usr/src.
I’m going to talk about the install process now so things are a little clearer later on. At this point, we can install the new world and kernel onto the client box from builder. The process is very simple.
mount /usr/src && mount /usr/obj cd /usr/src make installkernel installworld mergemaster cd umount /usr/src /usr/obj shutdown -r now "Applying system upgrade."
/etc/make.conf if you are using a custom
It seems pretty straight forward but there are two caveats: 1) installkernel
doesn’t work if
kern.securelevel > 0 and 2) mergemaster is an interactive
tool. We can get around 1) by setting
/etc/ rc.conf and rebooting 2) is a bit more interesting.
The problem with mergemaster is that it runs a comparison of your system configuration files and replaces those things that have changed while allowing you to merge the changes with your existing files. “How is this a problem?” you might ask. It’s a problem because you’re not there to do the merge. You can put mergemaster in automatic mode with the -a flag which “will leave all the files that differ from the installed versions in the temporary directory to be dealt with by hand.” That’s all well and good for any new files that are added but doesn’t help when part of the upgrade involves a bug fix to one of the system rc files, for example.
So, what do we do about it? Well, if you are staying within a single security branch, then you can probably get away with ignoring the changes that mergemaster finds. If you are upgrading to a newer release, 4.8 to 4.9, for example, or there is a security or bug fix in /etc then you should run mergemaster in interactive mode on each server. Note: There may be a way of dealing with those types of changes automatically but use caution. If you’re not sure, then do it by hand.
To make upgrades easier, I wrote a little sh script that I call upgrade.sh. When run without arguments, it disables kern.securelevel in /etc/rc.conf, puts a short script in /usr/local/etc/rc.d and reboots the server. The rc script, called 000.upgrade.sh by default, will call upgrade.sh with one argument: install.
When called with the install argument, upgrade.sh will:
- mount /usr/src and /usr/obj
- install the new world and kernel
- run mergemaster -a
- unmount /usr/src and /usr/obj
- re-enable kern.securelevel
- remove 000.upgrade.sh
- and, reboot the server.
At the end, your server should be running the new version of FreeBSD, complete with your custom kernel.
The autoroot User
Now we’re going to add a user, autoroot, and give it the permissions it needs to run upgrade.sh.
We need to be careful with autoroot because we are going to be giving it a bit of power that can be dangerous in the wrong person’s hands. The first step to take to secure autoroot is to disable password logins. (This is done with the -w no option.) This will prevent anyone from logging in as autoroot without the DSA key.
pw user add autoroot -m -c "Auto root" -w no
We’ll be using sudo to give autoroot the permissions it needs. (If don’t have
sudo installed already, you can install it from security/sudo in the ports
collection.) The second step is to add autoroot to
# Begin AUTOROOT setup User_Alias AUTOROOT_USERS = autoroot Cmnd_Alias AUTOROOT_CMDS = /home/autoroot/bin/ Defaults:AUTOROOT_USERS !lecture AUTOROOT_USERS ALL = NOPASSWD: AUTOROOT_CMDS # End AUTOROOT setup
We’re going to all autoroot to run any command in
/home/autoroot/bin/ as root.
This will give us some flexability with the commands autoroot can run without
having to edit sudoers. You can allow autoroot to run other commands by
creating symlinks in
I want to draw your attention to the
NOPASSWD option. This is required because
it’s going to be hard for autoroot to enter a password each time sudo asks for
it. Besides, autoroot doesn’t have a password. :-) This is one of the reasons
you want to be careful with autoroot because, if someone does log in as
autoroot, they do not have to enter a password to run your scripts.
Finally, add your SSH key to
~autoroot/.ssh/authorized_keys so that you can
log in with your private key.
cat id_rsa.pub >> /home/autoroot/.ssh/authorized_keys
Upload the update.sh script into
~autoroot/bin/. I’ll show you an easy way to
do this later.
#!/bin/sh BUILDER=192.168.1.1 RC_CONF=/etc/rc.conf RC_D=/usr/local/etc/rc.d RC_SCRIPT=000.upgrade.sh case "$1" in install) mount /usr/src || mount $BUILDER:/usr/src /usr/src mount /usr/obj || mount $BUILDER:/usr/obj /usr/obj cd /usr/src echo "make installkernel installworld" echo "mergemaster -a" cd umount /usr/src umount /usr/obj sed 's/^kern_securelevel_enable="[Nn][Oo]"/kern_securelevel_enable="YES"/' $RC_CONF > $RC_CONF.new mv $RC_CONF.new $RC_CONF rm $RC_D/$RC_SCRIPT shutdown -r now "Booting to new kernel." ;; *) sed 's/^kern_securelevel_enable="[Yy][Ee][Ss]"/kern_securelevel_enable="NO"/' $RC_CONF > $RC_CONF.new mv $RC_CONF.new $RC_CONF echo '#!/bin/sh' > $RC_D/$RC_SCRIPT echo 'echo " Upgrading"' >> $RC_D/$RC_SCRIPT echo 'case "$1" in' >> $RC_D/$RC_SCRIPT echo 'start)' >> $RC_D/$RC_SCRIPT echo " $0 install" >> $RC_D/$RC_SCRIPT echo " ;;" >> $RC_D/$RC_SCRIPT echo '*) ;;' >> $RC_D/$RC_SCRIPT echo "esac" >> $RC_D/$RC_SCRIPT chmod +x $RC_D/$RC_SCRIPT shutdown -r now "Preparing for upgrade." ;; esac
Now that we’re done with the servers, it’s time to setup the control box. This is the box that you will be issuing the autoroot commands from. It can be any box that has ssh access to your servers. The builder server might be a good choice. It should not be a production server because you’ll be opening quite a few ssh processes and it can bog down the box.
I use dcmd to issue the autoroot commands on each client. You can get dcmd from SourceForge.
This file contains the list of hosts to run the commands on and the user to
them as. Each entry is formatted as
[email protected]. We’ll use the autoroot user
that we setup earlier.
[email protected] [email protected] …
Now that dcmd is installed, let me show you an easy way to upload upgrade.sh.
We can use dscp to upload the script to every server listed in dcmd.hosts. The
 will be replaced with the entries in dcmd.hosts.
dscp upgrade.sh :bin/
Performing the Upgrades
From the control box, you can issue the upgrade command like so.
dssh sudo bin/upgrade.sh