Sugioarto

2015-08-21

Managing FreeBSD Jails Manually

If you are tired of using frameworks that complicate your server management and the implied restrictions that often are imposed to make everything seem easy-to-do, this is the right article for you as a FreeBSD Jails manager.

This article is an adaptation of the original FreeBSD Handbook section about managing multiple jails. It tidies up some directories and makes most things work out-of-the-box, so that you don't need to ask yourself why things don't work like you expect.

It is assumed that you want to install all your jails in the directory /jails. You should also compile world which should match your jail host. The directory where the sources are is /usr/src here.

Disclaimer

Please watch your step! I don't want to be held responsible for damaged data or hosts or whatever. You need to understand parts of the jail concept and you need to know that jails look exactly like you main host. Deleting a directory on your main host or misconfigure it will upset you and your users.

Think before typing and check that I haven't done mistakes while writing this article.

I repeat once again: these are very low-level operations and require an experienced administrator.

Remove old rootfs master

You can skip this procedure, when you have never installed any previous jails with the method described in this article.

When updating the mroot tree, all jails need to be stopped and their filesystems need to be unmounted. It is not possible to update the jail's rootfs while it is running.

The mroot is moved in a safe place, so you can restore it when the updated jail does not work instantly and people are impatient about service downtimes.

cd /jails chflags -R 0 mroot.bak rm -rf mroot.bak mv mroot mroot.bak

Bootstrap rootfs master

In this section the mroot master rootfs is created. It is not used for a specific jail, but for all jails at same time, and for this reason, it should be kept equal for every jail which you will install on your system. mroot is big, but since it is needed only once for all you jail installations, the concept overall is very lightweight.

This mroot will be mounted as a read-only unionfs in each jail. Inside this read-only jail rootfs, there is the special mountpoint /s which is used for writable files and configurations. The mroot is prepared here to point to /s using softlinks for every writable directory.

cd /jails mkdir mroot cd /usr/src make installworld DESTDIR=/jails/mroot cd /jails/mroot chflags -R 0 var rm -rf etc var boot rmdir root tmp mkdir s ln -s s/etc etc ln -s s/home home ln -s s/root root ln -s s/tmp tmp ln -s s/var var cd usr mkdir ports rmdir local obj ln -s ../s/usr-local local ln -s ../s/tmp/obj obj ln -s ../s/home home

Remove old skel master

You can also skip this section, if you have never installed jails with the method described in this article.

cd /jails chflags -R 0 skel.bak rm -rf skel.bak mv skel skel.bak

Make a new skel master

skel is the master copy for the contents of /s. You should make one for each jail that you want to use.

skel is not big, but you make copies of it for individual /s directories and since /usr/local of each jail is inside /s, it can get huge, if you install many ports there.

Please notice that you need to replace the placeholder string DNS-SERVER-IP by your DNS server IP and example.org by a valid domain name that you want to use.

Also please notice that the jail created from this template will make it start its own sshd (SSH daemon), which allows remote logins. Don't add it to /etc/rc.conf, if you don't like it. Most people might want it, though.

cd /jails mkdir skel cd skel mergemaster -t /jails/skel/var/tmp/temproot -D /jails/skel -i rm -rf .cshrc .profile bin boot dev lib libexec media mnt proc rescue sbin sys usr mkdir usr usr-local home distfiles packages wrkdir tmp/obj cd usr; ln -s ../usr-local local; cd .. echo 'sshd_enable="YES"' >> etc/rc.conf echo "IGNORE_FILES='.cshrc .profile COPYRIGHT /boot/device.hints'" > etc/mergemaster.rc cp ../skel.bak/etc/resolv.conf etc/resolv.conf

If the last command fails, because you haven't had a previous resolv.conf file, make a new one with the proper settings:

cd /jails/skel/etc echo "domain example.org" > resolv.conf echo "nameserver DNS-SERVER-IP" >> resolv.conf

Make a new jail

This is simple, if you understood the last section about skel. It is just needed to copy skel to a directory which you can mount to /s later when you mount your jail filesystem tree.

I will call the jail XXX here, so you can easily see where it needs to be replaced in the commands you type.

You need to copy the skel contents (which is a template for the writable jail contents) to XXX-s. Please don't mix it up with XXX here which is for the read-only rootfs and needs to stay empty for now.

cd /jails cpdup skel XXX-s

Prepare jail filesystems

Perhaps you can see clearer how it is managed, when you look at the mount hierarchy for our new jail called XXX.

We mount mroot (as told above) read-only in the specific jail rootfs. Then at its specific /s directory, we mount the writable contents that we copied from the template skel.

We also need the ports tree from our jail host. It saves space to make an union mount, which is also read-only. Though, you will need some hacks in /etc/make.conf to make it work (see further below).

We also add a /usr/src for the jail. You can also add it with "noauto", if you don't like it to have /usr/src always mounted in your jail.

In /etc/fstab, add:

/jails/mroot /jails/XXX nullfs ro 0 0 /jails/XXX-s /jails/XXX/s nullfs rw 0 0 /usr/ports /jails/XXX/usr/ports nullfs ro 0 0 /usr/src /jails/XXX/usr/src nullfs ro 0 0

Mount jail filesystems

Since we don't want to restart the system, we mount the filesystem tree manually.

mkdir /jails/XXX zfs create pool/jails/XXX-s mount /jails/XXX mount /jails/XXX/usr/ports cd /jails cpdup skel XXX-s mount /jails/XXX/s

Configurations for a new jail

Example /etc/make.conf

In case you don't use package build systems like Poudriere and use portmaster or similar local build tools, you will need to make these settings that should be modified for your jails (except for the first three lines here with the directory paths).

DISTDIR=/s/distfiles WRKDIRPREFIX=/s/wrkdir PACKAGES=/s/packages WITH_PKGNG=yes MAKE_JOBS_NUMBER=3 WITHOUT_X11=yes WITHOUT_GNOME=yes

Example /etc/jail.conf

This is the configuration for the jail host which makes /etc/rc.d/jail start XXX work properly.

Replace JAIL-IFACE with the jail network interface that the jail is going to be bound on.

Replace IP4-ADDRESS-OF-JAIL with the IP4 address you would like to assign to the jail.

You need to repeat the XXX { ... } section for every jail you have created.

exec.start = "/bin/sh /etc/rc"; exec.stop = "/bin/sh /etc/rc.shutdown"; exec.clean; mount.devfs; mount.fdescfs; allow.sysvipc; path = "/jails/$name"; interface = JAIL-IFACE; XXX { host.hostname = "XXX.example.org"; ip4.addr = IP4-ADDRESS-OF-JAIL; enforce_statfs = 1; }

Jail startup

You can start your jail now.

/etc/rc.d/jail start XXX

Update a jail

In case you have not created a jail by copying skel, but updating it, you need to run mergemaster inside it to update your system configurations. Remember that mroot (rootfs) is now up-to-date, but the jail's configurations need to be updated, too. You also need to start the jail.

In case you decided to make your jail's /usr/src mounted noauto, you will need to mount /jails/XXX/usr/src first and after the procedure umount /jails/XXX/usr/src it again.

jexec XXX tcsh etcupdate exit

Final setup for new jails

In case you have just updated jails, you can skip this section.

You should verify with jls that your jail is really running. Use the JID of the jail in the jls listing to enter a jail with root shell.

Here, you setup the timezone, the root password and add your first user (usually you, and you would like to add yourself to the wheel group).

jls jexec JID tcsh tzsetup passwd adduser etcupdate extract

Summary

This guide shows how to setup a very lightweight FreeBSD jail with enhanced security, because of read-only rootfs. I have used this concept for years now and I am sure that it works very well (I checked it on FreeBSD-10.2).

Email me mistakes, if you find any. There might be some and I hope they are not fatal.