Thursday, August 27, 2009

Making Gentoo Even Harder: 64-bit Hardened Gentoo

As I mentioned in my last post, I recently upgraded my Gentoo laptop to use the hardened toolchain. I put Hardened Gentoo, with SELinux, on my servers at work, but those are all configured with the base system profile (selinux/2007.0/x86/hardened), which is stable but kinda old. With my personal laptop, I was a bit more daring, and grabbed the hardened-overlay.

The gory details of why there is a hardened-overlay in the first place can be found at the home page of the overlay -- particularly the bug report and forum discussion linked from there. But to summarize:

With gcc 4 and glibc 2.4, there were major changes in how the hardening features (PIE/PIC, stack-smash protection, etc) worked. The Gentoo hardened and toolchain teams had things working with the previous versions, but the effort involved in updating everything drove more than one developer to the brink of insanity. The end result was that gcc 4 was masked off on all of the hardened profiles in Gentoo, leaving us stuck with gcc 3.4 while the rest of the world moved on.

Fast forward a few years, and a some interested Gentoo users (Xake, Zorry, a couple others) picked up the where previous devs had left off, and published the "experimental" hardened toolchain overlay. The overlay, which was just recently moved to, renamed to hardened-development, and added to layman, contains updated ebuilds for gcc and  glibc that build everything with SSP + PIE. So, having had good results using this overlay on my x86 desktop in the past, I decided to give it a try on my amd64 laptop.

The biggest problem with using this overlay is that development and updates are outpacing the documentation -- not a bad problem to have in general, but it makes finding up-to-date information take some work. The most valuable resources I used during my upgrade were:
Fortunately, while the process make take a long time, its relatively painless and trouble-free. I was able to switch to the hardened toolchain and rebuild over 800 packages with only 3 failures. Here's what to do:

Install the Overlay

First things first: you need the overlay. If you don't already have layman installed, you should do that, then add hardened-development to your local overlays:
root@platypus ~ # emerge layman
root@platypus ~ # layman -L
root@platypus ~ # layman -a hardened-development
root@platypus ~ # echo 'source /usr/local/portage/layman/make.conf' >> /etc/make.conf
If you're using a 2.2+ portage version, there are a few more steps to make things work properly. Because this overlay is changing core system behavior, it must override some of the base eclasses, which causes problems with the pre-generated metadata cache that normally comes with a portage tree sync. What we need to do, instead, is to ignore the cached metadata completely and let portage re-generate it locally. We also need to tell portage that the overlay's eclasses take priority over the ones in the base tree.
root@platypus ~ # echo "[DEFAULT]\neclass-overrides = hardened-development" >> /etc/portage/repos.conf
root@platypus ~ # echo "[DEFAULT]\neclass-overrides = hardened-development" >> /etc/portage/repos.conf
root@platypus ~ # echo "[gentoo]\neclass-overrides = hardened-development" >> /etc/portage/repos.conf
root@platypus ~ # echo "[gentoo]\neclass-overrides = hardened-development" >> /etc/portage/repos.conf
root@platypus ~ # rm -rf /usr/portage/metadata/cache
root@platypus ~ # echo 'PORTAGE_RSYNC_EXTRA_OPTS="--exclude=/metadata/cache"' >> /etc/make.conf
After you do a sync of you portage tree and layman overlays, you'll want to run "emerge --regen" to force a rebuild of the dependency cache. If you don't, this will happen automatically the first time portage needs to look something up, and it can take a while.

Switch Profiles

With the overlay installed, you now have working ebuilds for gcc 4 and glibc 2.10. (You'll note that the references above sometimes refer to gcc 4 as being in the testing branch of the overlay -- this is no longer true.. Now, we need to switch to a profile that has the hardened USE-flag enabled by default. For the amd64 arch, this can be confusing. For example, my eselect profile list includes several likely candidates:
[11] hardened/amd64
[12] hardened/amd64/multilib *
[20] hardened/linux/amd64
In this case, both #12 and #20 support multiple ABIs (the "multilib" part), while #11 does not. (Note: your #'s may change depending on what profiles are in your tree). However, #12 seems to provide a better environment for x86 builds, so I'd suggest going with either #11 or #12, for no-multilib or multilib, respectively.

Unmask Packages

Whichever profile you pick, the gcc and glibc versions we want will have been masked off, since the ones in the main portage tree don't work. So we have to unmask them, as described in the how-to:
root@platypus ~ # echo "=sys-devel/gcc-4.4*" >> /etc/portage/package.unmask 
root@platypus ~ # echo "=sys-libs/glibc-2.10*" >> /etc/portage/package.unmask
A word here about the stable vs. unstable keywords here. All of the packages in the overlay are keyworded. If you are using only stable packages on your system, you will need to unkeyword them by adding them to package.keywords. In the past there were a slew of other packages that you also needed to un-keyword, listed in the How-To. I don't think any of those apply now, but things could change in the future. Personally, I strongly discourage people from playing mix-and-match with stable and unstable packages. If you're using this toolchain, you're already way beyond the realm of stable and supported, so I'd recommend just adding ~amd64 into your ACCEPT_KEYWORDS and being done with it.

Rebuild The Toolchain

Now, you're ready to rebuild the toolchain with the new hardened features. This step will prepare our build environment for when we rebuild the entire system later on, so it must be done first.
root@platypus ~ # emerge -1 gcc-config linux-headers binutils gcc glibc
Let this run to completion, then use gcc-config to select your new compiler. Your gcc-config -l output will show four options for your hardened compiler version: the "normal" one, a "vanilla" option, and "nopie" and "nossp" options. You want the normal one to be your default, but later on we will switch to the nopie version for a couple of items.

Install A New Kernel

To really the get most out of your hardened toolchain, you'll want to use a kernel that knows how to enforce the rules on what a properly hardened application shouldn't be doing. The hardened-sources kernel package includes patches for, among other things, all of the PaX features developed by the hardened team.
root@platypus ~ # emerge hardened-sources
root@platypus ~ # eselect kernel list 
[1]   linux-2.6.29-hardened
  [2]   linux-2.6.30-gentoo-r5 *
root@platypus ~ # eselect kernel set 1
root@platypus ~ # cd /usr/src/linux
root@platypus ~ # make menuconfig 
The excellent PaX Quickstart Guide, Section 4, will give you a list of the PaX options that you want to enable on your system. There are only a few changes you need to make to the options listed there, mostly because we're on an amd64 architecture:
Under PaX Control --->
[ ]   Use legacy ELF header marking 
You don't really need this, since you'll be rebuilding everything from scratch with the new ELF header markings. This would be mostly used if you had old binaries marked using an earlier PaX tool -- which you won't.
Under Non-executable pages --->
[ ]   Segmentation based non-executable pages  (not available)
[*]   Disallow ELF text relocations

Under Address Space Layout Randomization --->
[ ]     Randomize ET_EXEC base (not available)
Here we see some of the benefits of amd64 shining through. AMD64 and Intel's x64 CPUs natively support the no-exec bit on memory pages. Under x86, we had two options for non-executable memory: per-page (PAGEEXEC) and per-segment (SEGMEXEC), each with their own performance penalties. Under amd64, the per-page option uses the CPU's native support, with basically zero overhead, so there's no point in offering the other option.

The ABI for the AMD64 architecture also demands that all binaries be position-independent (PIE or PIC). The "randomize ET_EXEC" support is only used by position-dependent executables, so that option isn't even available to us. In addition, the only reason you'd want to allow text relocations was for position-dependent libraries that require them to load properly. Again, we don't have any of those, so we can turn on that option and eliminate one more avenue of attack.

After you build and install this new kernel, make sure you keep your old one hanging around for a while. Once you have a PaX-enabled kernel booted, incorrectly built binaries could be killed off when they try to load, if they try to do something PaX doesn't like. If this happens to include, say, init or bash, you could be in serious trouble. Set up grub to have your previous kernel handy before you reboot anything.

Rebuild Everything

The last step is the easiest, and longest. Now that our toolchain and kernel are all ready to go, we need to recompile everything on the system to be PIE with SSP enabled. So fire off the emerge command:
emerge --keep-going --emptytree @world
and go grab some lunch.

Dealing With Failures

You'll find that some packages just don't want to cooperate with the hardened toolchain. In most cases, the ebuilds themselves will handle any special setup that needs to be done, but a few packages are just flat out incompatible with the PIE/SSP features of the compiler. On my system, out of 803 packages, 3 of them died during the emerge process: dev-lisp/sbcl, media-video/mplayer, and app-office/openoffice. SBCL we dealt with last time by switching to CLISP, but it could also have been handled as described below.

If you have a package that just refuses to cooperate, you may need to switch back to a non-hardened compiler to build it, then tell the kernel that it's ok to let that particular binary do bad things. Taking mplayer as an example: some of the hand-coded assembler in the mplayer package doesn't fit within the restructions of a PIE binary (which reserves one more register than usual for its own purposes), so you get errors. (mplayer is a notoriously problematic application for hardened users because of all the hand-coded and optimized assembly it uses.)  To work around the problem, we compile mplayer using the "nopie" option of our compiler:
platypus ~ # gcc-config 2
* Switching native-compiler to x86_64-pc-linux-gnu-4.4.1-hardenednopie...
>>> Regenerating /etc/

* If you intend to use the gcc from the new profile in an already
* running shell, please remember to do:

*   # source /etc/profile
platypus ~ # source /etc/profile
platypus ~ # emerge -1 mplayer
platypus ~ # gcc-config 1
platypus ~ # source /etc/profile
Once the package is built, you may find that trying to run it just causes the kernel to kill it off, writing errors to your dmesg such as :
PAX: execution attempt in: 
PAX: terminating task: 

To prevent this, you need to tell PaX that this one program is allowed to do things it shouldn't be doing. For example, if we really wanted to stick with SBCL, it can be made to run under a hardened kernel by doing this:
root@platypus ~ # emerge paxutils 
root@platypus ~ # paxctl -pmr `which sbcl`
The flags that are needed depend on exactly what the binary is trying to do, but typically the -p and -m flags are the big culprits. The paxctl utility actually changes the ELf header in your binary, so its settings are persistent as long as you don't upgrade or reinstall. Of course, if you can find an alternative package that does the same thing without any PaX flags, that's usually a better choice.

In Conclusion

Now that you have a spiffy new Hardened Gentoo AMD64 laptop, its time to show off to your friend.  I suggest app-admin/paxtest in 'blackhat' mode, or this script from the trapkit blog (which requires dev-libs/elfutils).