Barking Iguana

New exploits are found just about every day in software both old and new. To combat this, software vendors release security updates which the Ubuntu team package up and use to release new, more secure packages of the software that you install.

It’s very hard to provide software updates for all versions of all software packages ever built for a platform like Ubuntu so the Ubuntu team produce releases of Ubuntu Linux which they offer to support for a known period of time. These support periods are currently one of two lengths depending on the release of Ubuntu you use. Long Term Support (usually noted by including “LTS” in the name) releases for the server are supported for 5 years after the release date. Regular releases are supported for 18 months. Beyond these windows support for the release is dropped and you won’t be able to upgrade or update any packages without a lot of work, so it’s important to upgrade before support for your release is stopped.

The current commonly used releases, their release date, and their end of support window are as follows:

Version	Name	Release Date	Support Ends
10.04 [LTS]	Lucid Lynx	April 2010	April 2015
9.10	Karmic Koala	October 29 2009	April 2011
9.04	Jaunty Jackalope	April 23 2009	October 2010
8.10	Intrepid Ibex	October 30 2008	April 2010
8.04.4 [LTS]	Hardy Heron	January 28 2010	April 2013
8.04.3 [LTS]	“	July 16 2009	April 2013
8.04.2 [LTS]	“	January 22 2009	April 2013
8.04.1 [LTS]	“	July 3 2008	April 2013
8.04 [LTS]	“	April 24 2008	April 2013
7.10	Gutsy Gibbon	October 18 2007	April 2009
7.04	Feisty Fawn	April 19 2007	October 2008
6.10	Edgy Eft	October 26 2006	April 2008
6.06.2 [LTS]	Dapper Drake	January 21 2008	June 2011
6.06.1 [LTS]	“	August 10 2006	June 2011
6.06 [LTS]	“	June 1 2006	June 2011
5.10	Breezy Badger	October 12 2005	April 2007
5.04	Hoary Hedgehog	April 8 2005	October 2006
4.10	Warty Warthog	October 26 2004	April 2006

Currently supported are 6.06, 8.04, 8.10, 9.04 and 9.10. Ubuntu 10.04 will be released in April.

Your responsibilities

There are two things that you as a server operator need to know how to do to stay up-to-date with the software on your server and make sure that it’s always supported. One is upgrading installed packages, and one is upgrading to the next release of Ubuntu. I’ll cover them both one at a time, but first we’ll do a little setup to make sure that both operations are nice and fast.

Using a package mirror

A very time consuming part of the update and upgrade process is downloading a large quantity of files from the servers that host the latest software packages. To make this faster Xeriom Networks provide a mirror of the software packages for 8.04, 8.10, 9.04 and 9.10. If you’re not hosted on Xeriom Networks (why not?) you should ask your current provider if they supply a package mirror for your release. If they don’t you should skip this section and hope that your connection is fast enough to cope.

Using the package mirror is simple and requires you to edit just one file. A nice simple editor is nano. Install it by connecting to your server by SSH and using sudo and apt-get to install it:

sudo apt-get install nano --yes

The contents of the file is determined by which release of Ubuntu you are running. Find out which release you run by typing this:

cat /etc/lsb-release

You should then match your release up with the appropriate box on this Wiki page: http://wiki.xeriom.net/w/XeriomUbuntuPackagesService

Copy the text from within only the box that’s matched with your release. Now we can edit the file to add the local package server:

sudo nano -w /etc/apt/sources.list

Delete all the existing lines in that file and replace them with the text you copied from the Wiki page. Now press CTRL and X and say that you do want to save the file.

Now we tell Ubuntu to refresh the list of software packages so it knows what’s available on the local package mirror:

sudo apt-get update

Congratulations, you’re now using the Xeriom Ubuntu package mirror.

Upgrading installed software

Keeping your software up-to-date is an important part of keeping your server secure but since these new packages may break existing functionality it’s best not to install them automatically. You should sit down and do this yourself, only applying the updates if they’re appropriate and necessary.

To update the software installed on your server you use a set of tools called apt – and specifically you use apt-get.

To make sure you get the latest updates you should first update your servers package database:

sudo apt-get update

Then you should ask it to upgrade your existing packages:

sudo apt-get upgrade

This will calculate everything that needs upgrading, show you a list of those packages and ask you if you want to continue. Most of the time this command will run smoothly and you’ll get the latest version of the software on your server, but you should check the list of packages that it will upgrade and be sure that you know what’s going to change before you let it complete.

Upgrading to the next release

Upgrading Ubuntu is a slightly more time consuming process with a small increase in risk because of the huge number of packages that will be upgraded. It will usually also require you to reboot your server since the kernel is likely to be upgraded so you should plan for a little downtime.

To upgrade you should use a package called update-manager-core. If this is your first time upgrading your Ubuntu release you may need to install it:

sudo apt-get install update-manager-core

Next configure it to target releases based on your preferred strategy. This is either lts, normal or never.

sudo nano -w /etc/update-manager/release-upgrades

Change the line that starts “Prompt=” to be whichever strategy you choose. For example, if I choose the lts strategy which should give me 5 years of support for each release I update to I’d enter “Prompt=lts” here. Now press CTRL and X and tell it that you want to save the file if it asks.

Now, before you upgrade, read the release notes for the version of Ubuntu that you’re going to upgrade to and make sure you understand all the current issues and caveats surrounding it.

Once you’re happy that you understand what you’re doing and you’ve scheduled a time to upgrade your server you can start the upgrade:

sudo do-release-upgrade

This will calculate all the packages that will be upgraded and ask you if you want to continue. Don’t just say yes – read the list of packages and make sure you understand what upgrading them means to your setup.

If it all goes pear-shaped

Sometimes things get messed up. Maybe the release wasn’t tested enough (rare these days) or perhaps the new release doesn’t support the same software as the one you upgraded from and you need that to run. If this ever happens we can create you a brand new image of the release you need as long as it’s still supported. Of course, your data won’t be on the new release so make sure your backups are up-to-date!

You've heard again and again that storing passwords in plain-text is a bad idea so now you store your passwords as MD5 or SHA1 hashes... if someone steals your password database your users passwords are safe, right?

Actually no, they're not. They're never totally safe. You can however make the amount of effort required to break into an individual account too large for all but the most dedicated (and longevous) attacker.

Unfortunately most of the web applications that I get a chance to examine don't take the chance to make their password storage more secure, which is a shame because it's really not difficult.

For completeness, let's start with the simplest possible way of storing passwords.

Plain Text Passwords

Anathema to account security, if your password database is compromised then none of your accounts have any protection. Congratulations, you have just given away the details of your entire userbase.

That's not the worst of it either, anyone listening to the traffic between your password database and your application could pluck the passwords out of the air - very few people secure their database connections using TLS or SSH. If you're on an unswitched network spying on something like MySQL traffic becomes as easy as running one command:

# tcpdump -l -i eth0 -w - src or dst port 3306 | strings

Any query such as SELECT users.id FROM users where password = 'foo' or results from a query like SELECT users.* FROM users will show up in plain text and you won't even know that your passwords have been stolen.

In a switched environment it's possible to trick the switch into sending you traffic (although this can be detectable). Depending on the switching hardware it's also possible that in the event of a failure the switch will fail open and start acting like you're on an unswitched network.

Simple hashed passwords

Hashed passwords are generally looked at as the solution to this problem. No passwords are stored in plain-text, and it's hard to guess a password that'll match a certain hash so even if the password database is compromised or snooped you don't need to worry too much.

That may have been true, but hashes for many common words, passwords and passphrases have already been calculated. Translating from those hashes back to a password that'll match is trivial. Remember, since the original password isn't stored, all you need to do is match the hash - the actual input doesn't matter.

How easy is it to break into an account that's protected by an MD5 hashed password? Let's say we had attacked a site and found the following table of usernames and passwords:

Username : Hashed Password
Alice    : a34bc26f864ed5f404eac5b7a20cd9aa
Bob      : 7a75a532aaab234ad4bd33ed67e67242
Malory   : 39579c8d4a536eb092f959b4a3d14aa8
Zebedee  : 57208d910b63e879d2bae3b3a5f8366d

All we'd need to do is take the hashed password and look it up in a Rainbow Table for the hash that we think we're using. Given the length of these hashes (32 hexadecimal characters) it's pretty likely that they're MD5 hashed passwords so we could use something like GData to look for these hashes in an MD5 hash table.

Username : Password
Alice    : alphabets
Bob      : ch1cken
Malory   : blue41
Zebedee  : ?????

Only Zebedee is safe, and that's only for two reasons: (1) He's a freaky little spring creature that can do magic things who has a magnificent moustache, and (2) no one has added his password or some value that matches the hash of it to the rainbow table... yet.

Rainbow tables exist for several hashing algorithms including the shown MD5 and SHA1. If the hash wasn't on the rainbow table then causing a collision for a specific account would cost only USD$2,000 and take about a day for MD5. I'm not sure how much this would cost using SHA1.

Multiply hashed passwords

Rainbow tables take quite some time to populate and that time can be made longer by requiring that the hashing function is applied to the data several times before the value is taken.

MD5(alphabets)                        = a34bc26f864ed5f404eac5b7a20cd9aa
MD5(a34bc26f864ed5f404eac5b7a20cd9aa) = dd3f1bf5a36529705d08fe50b966d41a
MD5(...)                              = ...
MD5(...)                              = b5fdbbd055fcbfd3958a28f15661aea0

Each step takes a certain amount of CPU time to complete so the cost of generating a rainbow table that matches these hashes is higher, but CPU time is cheap these days. The advantage here is that the attacker doesn't know how many times you've applied the hash function unless they also have your code. Unfortunately that's reasonably easy to brute-force by starting at the hash and working backwards through the generated rainbow tables until you get to a value that allows the account to log into your site. After that magic number has been established you have just a few days before the rest of your accounts are compromised.

Peppered hashes

Rainbow tables can be calculated reasonably fast and while they're not hugely cheap, they're no longer exactly prohibitively expensive. How can we make it so that this is not a viable attack vector?

The rainbow tables don't enforce any rules on their input - they're maps of hashes to the input that generates them. If we require that each password contain a little data - a piece of spice (let's call it pepper) - that we specify then we can make limit the usefulness of the rainbow tables... A new rainbow table has to be generated where each input to the hash function has this pepper.

The pepper is stored in the authentication code in your application and never gets to the database except as part of a hash. In this way it behaves much like the magic number in the multiply hashed passwords above. Similarly, this one has the disadvantage that once the pepper has been discovered it can be used for other values.

Someone could - and if they're intent on it, will - calculate a rainbow table with a decent hit rate given time, but this does mean that there's no off-the-shelf rainbow table if you pick a strong, unique enough pepper.

Spicy hashes

By combining the pepper and the multiple runs of the hash function we could make the attacker have to guess two elements - the number of times the hash function is applied and the pepper.

pepper = ...aliesc3ifCTAasd4$af...
MD5(pepper + password)    = ...b5f34...
MD5(pepper + ...b5f34...) = ...ea28c...
MD5(pepper + ...ea28c...) = ...

SELECT users.id FROM users WHERE hashed_password = ...

I'm not sure that this actually buys anything over applying the hash function lots of times, but it sure looks pretty and I really wanted to make some sort of pun about using lots of pepper to make hashes spicy. Sorry.

Salted hashes

The pepper or the multiple-hash approach use only one value for the entire database - either the number of times the hash function is run on the input or the value that's added to the password. What if there were several values? What if there was one for each account? A small value that's unique to each account that's added in the same way as the pepper above - a salt for the hash.

In this case the rainbow table that needed to be used for an individual account would be of no use when it came to guessing the password for the next account.

Sounds awesome but where do we store the salt? I quite like storing it in the first few characters of the hashed password field, though you may prefer to store it in a separate column in the password database.

That's right, store it in the password database. Sounds like it'd make it easier to crack, right? Well, all this salt tells the attacker is that the password is somehow combined with this salt to generate the hash. The how is still hidden in your application code, and a valid password is still several iterations of rainbow table generation away... and that's for each individual account in your password database.

Safe now?

Hell no. With a decent combination of the above using strong salts, peppers, and a decent number of hashing calls you've made it unlikely that someone who has stolen your password database can access your users' accounts using it. That doesn't mean that your users will pick sane or safe passwords, that your system is free of bugs, or that there aren't other ways into your system or of finding your users passwords.

If you manage anything but the simplest of setups you've probably got an LDAP server setup providing directory services to your network. If you don't you should probably stop reading now ;)

Authenticate using LDAP

The first step to getting your Rails application authenticating using LDAP is to get Apache to authenticate all requests before they reach the application. This stuff is tricky and Apache already has a rather lovely module, mod_authnz_ldap, that does all the heavy lifting for us.

<VirtualHost 193.219.108.xxx:443>
  # I've used port 443 above because I'm dealing with passwords.
  # [...snip...]
  <Directory /var/www/foo.example.com/current/public>
    AuthType Basic
    AuthName "Foo Application Control Panel"
    AuthBasicAuthoritative off
    AuthBasicProvider ldap
    AuthLDAPUrl ldap://ldap.example.com/ou=people,dc=example,dc=com?userid?one
    Require valid-user
  </Directory>
  # [...snip...]
  # Your normal Rails HTTP configuration goes here
</VirtualHost>

Look up the user in Rails

Okay, so any request that hits your application is now authenticated against your LDAP directory. Next, tell Rails to look for the user. For authentication I wrote a rather funky (if I do say so myself) mixin, Xeriom::Acts::ProtectedSystem.

module Xeriom # :nodoc:
  module Acts # :nodoc:
    module ProtectedSystem # :nodoc:
      def self.included(base)
        base.send(:extend, ClassMethods)
      end

      module ClassMethods
        def acts_as_protected_system
          include InstanceMethods
          send(:before_filter, :ensure_user_is_logged_in)
          send(:helper_method, :current_user)
          send(:helper_method, :logged_in?)
        end
      end

      module InstanceMethods
        def ensure_user_is_logged_in
          if !logged_in?
            authenticate_user
          end
        end

        def logged_in?
          !current_user.blank?
        end

        def current_user
          @current_user ||= User.find_by_id(session[:user_id])
        end

        def current_user=(user)
          @current_user = user
          session[:user_id] = user.blank? ? nil : user.id
        end

        def authenticate_user
          authenticate_or_request_with_http_basic("Protected Area") do |username, password|
            # Lock your application servers down to listen to only
            # the web tier or this will kick your ass.
            send(:current_user=, User.find_by_username(username))
          end
        end
      end
    end
  end
end

ActionController::Base.send(:include, Xeriom::Acts::ProtectedSystem)

Like the code licence section in the sidebar says: this code is totally public domain, just don't sue me. To use it just drop the code in your lib/ directory and then call acts_as_protected_system in your ApplicationController.

class ApplicationController < ActionController::Base
  helper :all # include all helpers, all the time
  protect_from_forgery # because CSRF sucks!
  acts_as_protected_system # lock the door
end

For bonus points...

If you found this article useful, give me some love over at Working With Rails.

Follow these simple instructions to block all traffic but SSH to your box. Once you have these rules running you can punch more holes as required.

sudo apt-get install iptables
sudo iptables -A INPUT -i lo -j ACCEPT
sudo iptables -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
sudo iptables -A INPUT -p tcp --dport ssh -j ACCEPT
sudo iptables -A INPUT -j DROP
sudo sh -c "iptables-save -c > /etc/iptables.rules"

If you'd like to save your current rules when you stop - or load the rules when you start the box, change your /etc/network/interfaces file so that it contains pre-up and post-down hooks to load / save the rules.

pre-up    iptables-restore < /etc/iptables.rules
post-down iptables-save -c > /etc/iptables.rules

If you're hosted at Xeriom Networks and would like to be monitored by the monitoring service there, allow ICMP Type 8 from monitor.xeriom.net.

sudo iptables -I INPUT 4 -s 193.219.108.245 -p icmp -m icmp --icmp-type 8 -j ACCEPT

Remember to save the new rules to the iptables.rules.

sudo sh -c "iptables-save -c > /etc/iptables.rules"