Upgrade

To begin, check the OS version:

$ cat /etc/*release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=8.04
DISTRIB_CODENAME=hardy
DISTRIB_DESCRIPTION="Ubuntu 8.04.0"

No sense starting with one that's not current. Let's fix that one first before proceeding. What's on the machine should be secure and updated and, and we should test that the package manager and its transfer mechanism works fine:

$ sudo apt-get dist-upgrade
$ sudo apt-get update
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=8.04
DISTRIB_CODENAME=hardy
DISTRIB_DESCRIPTION="Ubuntu 8.04.1 LTS"

Ok, updated.

Manifest

We should make sure we have a clean slate to start with, and also should make sure the package and files databases are current:

$ sudo apt-get install apt-file
$ sudo apt-file update
$ sudo updatedb

This will enable us to find things quickly.

Logging

We set the date appropriately and make sure to write everything sent to the logs for now. This is important for debugging a new host.

$ sudo cp /usr/share/zoneinfo/US/Pacific /etc/localtime
$ sudo ntpdate clock.psu.edu
$ sudo sh -c 'echo "*.debug -/var/log/messages" >| /etc/syslog.conf"
$ sudo /etc/init.d/sysklogd restart

Hostname

Our next order of business is to configure the host's name properly. Forward and reverse DNS should match the machine's configured IP in fully-qualified fashion. This IP will be used for binding services, and various things will hot work properly if we aren't set up correctly.

In our case, the machine was bootstrapped with an /etc/hosts entry only for 127.0.0.1 and a hostname which was not visible in DNS. Our IP was given to us in the Challenge, and from the DNS resolver on the machine as well as from my administrative host, it does look up fully in DNS:

$ host 1.2.3.4
4.3.2.1.in-addr.arpa domain name pointer 1-2-3-4.eng.corp.com.

$ host 1-2-3-4.eng.corp.com
1-2-3-4.eng.corp.com has address 1.2.3.4

The machine's hostname should be fully qualified so that domain information does not need to be hardcoded into e.g. SMTP configuration. If the short name is needed, the hostname command does have a --short option.

The full contents of the hosts file:

$ cat /etc/hosts
127.0.0.1       localhost localhost.localdomain
1.2.3.4         1-2-3-4.eng.corp.com 1-2-3-4

We should make this change permanent and affect it at runtime as well, so we can avoid a reboot:

$ sudo hostname 1-2-3-4.eng.corp.com
$ sudo sh -c "echo $(hostname) > /etc/hostname"

Networking

A few things should be touched on briefly to make sure the host is reasonably secure from a networking standpoint.

For one, the host machine should probably be changed to disable IPv6 if it is not in use, as it presents a security risk and is unnecessary in the common case. So that we don't revoke our own access, however, let's stop the SSH daemon from listening on IPv6 sockets:

$ grep -i address /etc/ssh/sshd_config
AddressFamily inet
ListenAddress 0.0.0.0

$ sudo /etc/init.d/ssh restart

Your current login could be using ipv6 as well, and other network services could be bound to ipv6 addresses. Verify there is nothing with:

$ netstat -nta | grep ^tcp6

This list should come up empty. If not, those services will need to be configured to bind only on v4 addresses.

Once we have disabled all services bound to those IPs, we must unplumb the protocol families in question:

$ ip addr | egrep "inet6|^[[:digit:]]" | awk '{print $2}'
lo:
::1/128
eth0:
ab12::3456:78ff:dead:6789/64
eth1:
ab12::3456:78ff:dead:6789/64

$ sudo ip addr del ::1/128 dev lo
$ sudo ip addr del ab12::3456:78ff:dead:6789/64 dev eth0
$ sudo ip addr del ab12::3456:78ff:dead:6789/64 dev eth1

Finally, assuming this was successful, we disable reinstallation of the module next time something tries to open a socket with that protocol family:

$ sudo sh -c 'echo blacklist ipv6 > /etc/modprobe.d/blacklist-ipv6'

I have never been able to figure out a way to do this on Linux without rebooting, due to the following issue:

$ lsmod | head -1; lsmod | grep ipv6
Module                  Size  Used by
ipv6                  313384  14

For a last verification, let's run a last check to make sure nothing is listening that we aren't aware of:

$ netstat -lut
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State
tcp        0      0 *:ssh                   *:*                     LISTEN
tcp        0      0 localhost:smtp          *:*                     LISTEN

The only things listening are the SSH server and the SMTP server. We will deal with SMTP later since that's part of the instructions for setting up the server, but for now let's get rid of this now so we don't forget:

$ sudo fuser -avn tcp smtp,,
                     USER        PID ACCESS COMMAND
25,,/tcp:            Debian-exim  13229 F.... exim4

$ sudo apt-get -qq remove exim4\*
(Reading database ... 12897 files and directories currently installed.)
Removing mailx ...
Removing exim4-daemon-light ...
 * Stopping MTA [ OK ]
Removing exim4-base ...
Removing exim4-config ...

$ sudo fuser -avn tcp smtp,,
                     USER        PID ACCESS COMMAND
smtp,,/tcp:

If other services were running on this host, we would figure out which they were using a similar procedure, and either stop them with their /etc/init.d and make permanent with an update-rc.d invocation or remove the package.

The other points to hit are: make sure IP forwarding is disabled and make sure source address validation is enabled. Ubuntu appears to have sane defaults here (see ip_forward and rp_filter in /etc/sysctl.conf).

AppArmor

WARNING this section does not apply to the VPS in question, but it came up when I did my "dry run" in an Ubuntu 8.04 KVM instance on my laptop, so I leave it in the document.

Even coming from the SELinux world, I don't usually enable SELinux: it's overkill for most scenarios, changes are a pain, the system is somewhat complicated, and relabeling is slow. I personally think the complexity of SELinux is a little bit more trouble than it is worth for all but the most sensitive applications. Stay on top of your machines, keep them updated, use best practices that have been in use forever (like chroot), and you will minimize your risk without overly complex solutions like SELinux MACs.

Process isolation using compartmentalization technologies such as chroot or containers (lightweight) or paravirt/fullvirt (heavy) can do the job here as well; also there are modern thwarting techniques like non-executable stacks, randomized runtime vmap offsets, readonly system mounts and similar technologies that raise the bar without being overly complicated.

The Ubuntu AppArmor looks interesting insofar as it's a little bit easier to administer that SELinux, but still suffers from the manageability problems of SELinux and requires administration.

For example, when I added the LDAP database in a different directory than usual, AppArmor would not allow me to proceed, and it took me a bit of time to figure out because they failing system call (revealed by strace) worked fine with a test C program I wrote that did the exact same thing. Once I made syslog more verbose I saw the errors.

To eliminate any future unforeseen headaches, I removed AppArmor protection in order to get the thing moving and eliminate this issue:

$ sudo /etc/init.d/apparmor stop
$ sudo update-rc.d -f apparmor remove

But leaving AppArmor running might be worth considering for production. I'll leave these comments here since I took the time to write them and it may or may not be interesting to those evaluating me.