Sugioarto

2018-01-01

NFSv4 and Kerberos on FreeBSD

This is a HowTo that describes a NFS setup based on Kerberos (Heimdal) that enhances the security while sharing NFS filesystems. While running an unprotected NFS server on a network can be a risk, when a NFS server is accessible and users have physical control over their workstation, combining NFS with Kerberos can protect the NFS-shared data against access from unauthorized users.

This HowTo assumes that you run two or three hosts. This is a small sample setup, but it should give you an idea how to expand it for your specific use case.

A two-hosts sample setup used here is:

The three-hosts setup would be:

I added the latter one for better understanding how the host keys are managed. You'll see later why this might be confusing without considering separate hosts. If you decide to use the two-hosts setup, you will need to follow KDC and NFS server setup for the server host and merge the configurations, e.g. /etc/rc.conf, manually.

Notes about security

Please check the security measures, after you have followed the HowTo. Many steps and hints here are not documented at all and I make errors, too. This is especially important, because you probably want to secure sensitive data with the help of Kerberos.

Prerequisites

There are simple things that can make your life hard. Well, they have made mine hard, because I haven't understood them for over a year. And you can avoid these kinds of traps and save some time by reviewing your host configuration.

/etc/hosts on all servers and workstations involved

All kinds of components rely on identifying the hostnames. Please make sure that your hosts have exactly one hostname that is the FQDN. Please don't use any aliases.

Example of a correct host entry:

192.168.0.22 nfs.sugioarto.com

If you do this wrong, it will result in errors like this (you check and debug them by running gssd in verbose mode gssd -hvd):

gssd_acquire_cred: done major=0x70000 minor=0

The 7 in this code is not documented, but it means that the host key could not be loaded. Of course, it won't load correctly when the NFS server asks for a key with the wrong hostname.

Directory /var/heimdal on Kerberos KDC

If you ever have made some experiments with Kerberos/Heimdal, make sure that you tidy up. If you don't need your keys/tickets anymore then make sure that this directory is empty before continuing with this HowTo.

Packages

If you have replaced base openssl with libressl, like I have, you will notice that many packages that use GSSAPI will fail to install properly. I altered the options for GSSAPI to install Kerberos Heimdal from ports. This is not relevant for our purposes here, but it's sometimes nice to run more Kerberos services, if you already activate the KDC on a system.

Heimdal on KDC

Heimdal is a version of Kerberos used in the base system on FreeBSD. It is documented here. I'll describe the setup needed to run the NFS server and client here step by step.

/etc/krb5.conf for all hosts involved

This configuration tells clients how to find Kerberos services.

[libdefaults] default_realm = SUGIOARTO.COM [domain_realm] .sugioarto.com = SUGIOARTO.COM [realms] SUGIOARTO.COM = { kdc = server1.sugioarto.com admin_server = server1.sugioarto.com kpasswd_server = server1.sugioarto.com }

If you prefer your clients to use DNS, you can remove the [realms] part from this configuration and add the entries below to your DNS server. Here an example for unbound.conf. Insert these lines after your local-zone declaration:

local-data: "_kerberos.sugioarto.com. IN TXT SUGIOARTO.COM" local-data: "kerberos.sugioarto.com CNAME server1.sugioarto.com." local-data: "_kerberos._udp.sugioarto.com. IN SRV 01 00 88 server1.sugioarto.com." local-data: "_kerberos._tcp.sugioarto.com. IN SRV 01 00 88 server1.sugioarto.com." local-data: "_kpasswd._udp.sugioarto.com. IN SRV 01 00 464 server1.sugioarto.com." local-data: "_kpasswd._tcp.sugioarto.com. IN SRV 01 00 464 server1.sugioarto.com." local-data: "_kerberos-adm._tcp.sugioarto.com. IN SRV 01 00 749 server1.sugioarto.com."

/etc/rc.conf

You'll need some services to be started. Add these lines to your /etc/rc.conf:

kdc_enable="YES" kadmind_enable="YES" kpasswdd_enable="YES"

Start them now (as root).

/etc/rc.d/kdc start /etc/rc.d/kadmind start /etc/rc.d/kpasswdd start

Initialize the KDC

On KDC host:

kstash --random-key kadmin -l kadmin> init SUGIOARTO.COM Realm max ticket life [unlimited]: Realm max renewable ticket life [unlimited]: kadmin> q

The realm has been initialized.

Add your user principals

Without a valid user, you won't be able to access any single file on the file server, because you'll be always unauthenticated and consequently unauthorized. Even root cannot access files, if you don't generate a principal in Kerberos.

I'll just show how to add one single user (myuser), but it's trivial to add more. Make sure, you use system user names, so the credentials can be mapped to unix users.

kadmin -l kadmin> add myuser Max ticket life [1 day]: Max renewable life [1 week]: Principal expiration time [never]: Password expiration time [never]: Attributes []: myuser@SUGIOARTO.COM's Password: Verifying - myuser@SUGIOARTO.COM's Password: kadmin> q

Test KDC

As user myuser (generated above). Try to get a ticket:

kinit myuser@SUGIOARTO.COM's Password: klist

The last command should show you the ticket that you got by authenticating with the KDC.

If something fails, try to restart the KDC service with /etc/rc.d/kdc restart and/or revisit the steps above.

If everything is OK, you can destroy the tickets with:

kdestroy

You can see again with klist that they are gone.

Generate a principal for the NFS server service

The NFS server nfsd wants a principal when it starts. It is hardcoded and is constructed by using the prefix nfs/ concatenated with the hostname. For our three-host setup here it would be nfs/nfs.sugioarto.com. Since we agreed to have only one name for our host when KDC and NFS server are merged in our two-host setup, it would be nfs/server1.sugioarto.com, of course.

Here is how to generate the host key (for the three-host setup):

kadmin -l kadmin> add --random-key nfs/nfs.sugioarto.com Max ticket life [1 day]:unlimited Max renewable life [1 week]:unlimited Principal expiration time [never]: Password expiration time [never]: Attributes []: kadmin> q

Generate a principal for the NFS client

This is similar to above. Notice we use the prefix host/ here that we need to specify for the NFS mount with the parameter gssname=host on the NFS client later.

kadmin -l kadmin> add --random-key host/workstation.sugioarto.com Max ticket life [1 day]:unlimited Max renewable life [1 week]:unlimited Principal expiration time [never]: Password expiration time [never]: Attributes []: kadmin> q

NFS server setup

The NFS server consists of several components that need to interact together.

Add the host key to the NFS server

The gssd manages the tickets for the host and its services. It loads the keys from /etc/krb5.keytab. The nfsd asks for its ticket when it's started. We need to import it, but first we need it exported from the KDC.

Once again, we log in into KDC and export the NFS server service ticket to a file:

kadmin -l kadmin> ext_keytab --keytab=/tmp/nfs.keytab nfs/nfs.sugioarto.com kadmin> q

The file /tmp/nfs.keytab contains the secret host key. Please protect it adequately during transfers.

Copy this file over to the NFS server (e.g. with scp) and delete it from /tmp/nfs.keytab on the KDC host.

On the NFS server nfs.sugioarto.com, you will need to import the ticket into its /etc/krb5.keytab. The following command merges the ticket into this file, without destroying other imported tickets:

ktutil copy nfs.keytab /etc/krb5.keytab

Two-host setup

For the two-host setup, it's enough to import it locally, which is easier:

kadmin -l kadmin> ext_keytab nfs/server1.sugioarto.com kadmin> q

/etc/rc.conf

Put this in your NFS server /etc/rc.conf and replace IP with the IP that that NFS server needs to listen to:

nfs_server_enable="YES" nfsv4_server_enable="YES" nfs_server_flags="-t -h IP" nfsuserd_enable="YES" mountd_enable="YES" mountd_flags="-h IP" rpcbind_enable="YES" gssd_enable="YES" gssd_flags="-h"

Restart services

On the NFS server, after the ticket import, the services need to be restarted:

/etc/rc.d/gssd restart /etc/rc.d/rpcbind restart /etc/rc.d/mountd restart /etc/rc.d/nfsd restart /etc/rc.d/nfsuserd restart

/etc/exports

Minimal /etc/exports file for NFSv4 is this:

V4: / -sec=krb5p

Please be aware that security is now restricted to krb5p and everything else will fail. Please read the manual, if you don't want strict settings.

If you change /etc/exports, you'll need to restart mountd again:

/etc/rc.d/mountd restart

I decided to use krb5p as the most secure NFS mech. You can of course add more mechs. It makes sense to use sys for some exported filesystems. If you happen to export ZFS datasets with sys mech, please add the mech to the V4 line:

V4: / -sec=krb5p:sys

Also check the overall security twice, because this behavior is not documented well and exclusive krb5p should still be in effect to protect private shares from users faking a UID.

ZFS setup

You don't need to use /etc/exports for exporting the mountpoints, because ZFS has got commands to export datasets. The command to use is:

zfs set sharenfs='-sec=krb5p' pool/my-shared-dataset

You can also use additional /etc/exports options. There is one constraint. You can only export datasets that are mounted. If you try to export a dataset that is not mountable, even it's exported like above, you'll get cryptic errors on the client side.

In case you make a parent dataset mountable and mount it, the child datasets might disappear. You'll need to remount them again.

NFS client setup

These steps are needed for workstation.sugioarto.com which is the NFS client.

Import the host ticket

On the KDC server, we need to export the ticket for the NFS client workstation.sugioarto.com:

kadmin -l kadmin> ext_keytab --keytab=/tmp/krb5.keytab host/workstation.sugioarto.com kadmin> q

Copy the file /tmp/krb5.keytab over to workstation.sugioarto.com (e.g. with scp) and delete it from the /tmp directory on the KDC host.

On workstation.sugioarto.com, move the file to /etc/krb5.keytab and protect it:

chown root:wheel /etc/krb5.keytab chmod 600 /etc/krb5.keytab

/etc/rc.conf

Put this in your client's /etc/rc.conf:

nfs_client_enable="YES" nfscbd_enable="YES" nfsuserd_enable="YES" gssd_enable="YES" gssd_flags="-h"

Start the services:

/etc/rc.d/nfsclient start /etc/rc.d/nfscbd start /etc/rc.d/nfsuserd start /etc/rc.d/gssd start

/etc/fstab

You can mount filesystems on the client during late startup phase, when the network is available. Put this into /etc/fstab:

nfs.sugioarto.com:/my-shared-dataset /nfs/my-shared-dataset nfs rw,bg,soft,intr,nfsv4,tcp,sec=krb5p,gssname=host,late 0 0

Don't forget that the mountpoint needs to exist:

mkdir -p /nfs/my-shared-dataset

Mount it now:

mount /nfs/my-shared-dataset

Test it!

Try to access /nfs/my-shared-dataset as root.

cd /nfs/my-shared-dataset ls -l

You can see that it is not allowed. Great! Even root needs a ticket!

Log in as your user myuser that we have a valid ticket for.

cd /nfs/my-shared-dataset ls -l

It shouldn't work, either. Try again after getting a ticket. Type this as your user on workstation.sugioarto.com:

kinit

Setup Kerberos in PAM

A Kerberos login is nice because you get a ticket on login for free, but it has got its downside, because if the KDC is down, you'll get ugly delays when you need to log in.

There are two types of services that use PAM and for which you would like to generate tickets for transparently. system is a generic service that is imported almost everywhere. xdm is for the X display manager.

/etc/pam.d/system

# auth auth sufficient pam_opie.so no_warn no_fake_prompts auth requisite pam_opieaccess.so no_warn allow_local auth sufficient pam_krb5.so no_warn try_first_pass #auth sufficient pam_ssh.so no_warn try_first_pass auth required pam_unix.so no_warn try_first_pass nullok # account account required pam_krb5.so account required pam_login_access.so account required pam_unix.so # session #session optional pam_ssh.so want_agent session required pam_lastlog.so no_fail # password password sufficient pam_krb5.so no_warn try_first_pass password required pam_unix.so no_warn try_first_pass

/etc/pam.d/xdm

# auth auth sufficient pam_krb5.so no_warn try_first_pass #auth sufficient pam_ssh.so no_warn try_first_pass auth required pam_unix.so no_warn try_first_pass # account account required pam_nologin.so account required pam_krb5.so account required pam_unix.so # session #session required pam_ssh.so want_agent session required pam_lastlog.so no_fail # password password required pam_deny.so

Slim display manager

It should simply include system, but please check the file /usr/local/etc/pam.d/slim. It should look like this:

auth include system account include system session include system password include system