Verify database connections in long-running idle Rails processes
I've interfaced one of my xmpp4r bots with the Xeriom Networks control panel. I had intended to write a post about it to show how easy it is, but I've been beaten to it. I can, however, offer one piece of advice that will stop the bot dying after a few hours of idling: periodically verify your database connections.
RAILS_DEFAULT_LOGGER.debug "Launching database connection verifier"
Thread.new do
loop do
sleep 1800 # Half an hour
RAILS_DEFAULT_LOGGER.debug "Verifying database connections"
ActiveRecord::Base.verify_active_connections!
end
endAdding the above code to a script will stop database connections getting dropped (or, at least, will reconnect them if it happens).
To see it in action, add support@xeriom.net to your XMPP roster and have a chat. It's not hugely intelligent, but it does support a few useful commands.
Getting started with CouchDB: A simple address book application
I've recently installed CouchDB but, still being pretty new to this whole document store thing, don't really know what they can do or how to make CouchDB do it.
The best way to learn, of course, is to do. I've decided that I'll implement a simple address-book implementation.
Investigation and technology choice
Since CouchDB talks JSON I figure that I'll write the address book in Javascript and HTML, and because CouchDB includes a web server I'll serve the application from the same place I store the data. I'll call the file that contains that addressbook application addressbook.html.
Taking a peek at the CouchDB configuration in /usr/local/etc/couchdb/couch.ini I see that the document root for the web server can be found at /usr/local/share/couchdb/www - that's where the addressbook.html file will go.
I'll need a database to store people's contact details in. There's a pretty nice interface to do this at /_utils/ which is accessible using a web browser by pointing it at the CouchDB server's IP address and port.
CouchDB comes with a Javascript wrapper which can be found at /_utils/script/couch.js but it only talks to a local server and I'm accessing the page across the internet so I'll steal some code from it and change it to work for my setup.
Implementation
First off, create the database. Jump into the interface at /_utils/ and create a database called "addressbook". That's where we'll store our data.
The user interface is going to be a webpage using Javascript which makes things pretty simple. I'll whip up a really simple page to start with.
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<!-- The javascript will live in addressbook.js -->
<script src="http://aaa.bbb.ccc.ddd:5984/_utils/addressbook.js"></script>
<title>Address Book</title>
</head>
<body>
<h1>Address Book</h1>
<div id="addressbook">
<p id="loading">Loading... please wait...</p>
</div>
</body>
</html>Since I've been spoiled by ActiveRecord I want to be able to say something like var people = Person.find("all"); in my code and have it return all Person records. I also want to be able to say Person.find("123456-1234-1234-123456"); to find an individual person.
Person = {
// Push the implementation details of the database into a
// different object to keep Person clean.
//
database: AddressBook,
find: function(id) {
if(id == "all") {
return this.database.allCards();
} else {
return this.database.openCard(id);
}
}
}I've chosen to implement an AddressBook object that will abstract the details of database connection from the Person object. It will provide two methods, allCards and openCard(id). These methods talk to the CouchDB server and handle any and all data marshalling or other tricky bits and pieces.
AddressBook = {
// Change this to point to your own CouchDB instance.
uri: "http://craig-01.vm.xeriom.net:5984/addressbook/",
_request: function(method, uri) {
var req = new XMLHttpRequest();
req.open(method, uri, false);
req.send();
return req;
},
// Fetch all address book cards.
allCards: function() {
var req = this._request("GET", this.uri + "_all_docs");
var result = JSON.parse(req.responseText);
if (req.status != 200)
throw result;
var allDocs = [];
for(var offset in result.rows) {
var id = result.rows[offset]["id"];
var doc = this.openCard(id);
allDocs[allDocs.length] = doc;
}
return allDocs;
},
// Fetch an individual address book card.
openCard: function(id) {
var req = this._request("GET", this.uri + id);
if (req.status == 404)
return null;
var result = JSON.parse(req.responseText);
if (req.status != 200)
throw result;
return result;
}
}I push responsibility for parsing JSON off to another library. Luckily, Yahoo provide a rather nice JSON library that does just what I'm looking for - I don't have to implement it, but I do need to pull it into the webpage, and make it appear in the global namespace.
<!-- Add this to the head of addressbook.html -->
<script src="http://yui.yahooapis.com/2.5.2/build/yahoo/yahoo-min.js"></script>
<script src="http://yui.yahooapis.com/2.5.2/build/json/json-min.js"></script>
// Make YUI JSON available in the global namespace.
// Add this to addressbook.js
JSON = YAHOO.lang.JSON;The last piece of Javascript I need to show the address book is something to load all people from the address book and add them to the page. This uses window.onload hook which is bad, but for this little application is a quick and easy to kick off some code.
// This is horrible, I know, but it's just a simple example.
window.onload = function() {
var addressbook = document.getElementById("addressbook");
var personList = document.createElement("ul");
for(var offset in people) {
var person = people[offset];
var personNode = document.createElement("li");
var name = document.createTextNode(person.name);
personNode.appendChild(name);
personList.appendChild(personNode);
}
addressbook.removeChild(document.getElementById("loading"));
addressbook.appendChild(personList);
}That's it; the application is ready to go. Upload the addressbook.html and addressbook.js file to the document root of the CouchDB server, fire up your browser and navigate to http://aaa.bbb.ccc.ddd:5984/_utils/addressbook.html where aaa.bbb.ccc.ddd is the IP address of your CouchDB instance.
A blank page that says "Address Book" should greet you. Not very impressive, right? What went wrong? Actually, nothing went wrong. There's just no data in the database.
The interface that I pointed out before for browsing and creating databases can also be used to add documents to the database. Jump into it again, navigate to the addressbook database and add a document. When it asks you for an id, just leave the field blank: it'll create one automatically. Add a field to the document called name and click the little green checkbox beside the textbox, then double click on the value of the new field and set it to your own name in quotes eg "Craig Webster". Click the green arrow beside the textbox then click "save document", jump back to the address book and hit refresh. The new record should now show up.
Moving forward
I've shown how to retrieve data from CouchDB using Javascript, but currently the data still has to be input using the CouchDB interface. Watch this space for an upcoming article on manipulating the database using Javascript so we cna add cards to the addressbook.
Did this article help?
If this article helped you, I appreciate beer if you meet me, or recommendations at Working With Rails.
Installing CouchDB 0.8.0 on Ubuntu 8.04
CouchDB is a distrbuted document store which can be manipulated using HTTP. A more detailed introduction is available on the CouchDB site.
Some assembly required
Since CouchDB is still a fairly young project there are no packages available to install it on Ubuntu. There are rumblings which seem to indicate that Intrepid Ibis will have a package, but until then here's a quick-n-dirty way to get CouchDB running on Ubuntu 8.04.
sudo apt-get install automake autoconf libtool subversion-tools help2man
sudo apt-get install build-essential erlang libicu38 libicu-dev
sudo apt-get install libreadline5-dev checkinstall libmozjs-dev wget
wget http://mirror.public-internet.co.uk/ftp/apache/incubator/couchdb/0.8.0-incubating/apache-couchdb-0.8.0-incubating.tar.gz
tar -xzvf apache-couchdb-0.8.0-incubating.tar.gz
cd apache-couchdb-0.8.0-incubating
./configure
make && sudo make install
sudo adduser couchdb
sudo mkdir -p /usr/local/var/lib/couchdb
sudo chown -R couchdb /usr/local/var/lib/couchdb
sudo mkdir -p /usr/local/var/log/couchdb
sudo chown -R couchdb /usr/local/var/log/couchdb
sudo mkdir -p /usr/local/var/run
sudo chown -R couchdb /usr/local/var/run
sudo update-rc.d couchdb defaults
sudo cp /usr/local/etc/init.d/couchdb /etc/init.d/
sudo /etc/init.d/couchdb startLet others REST on your Couch
By default CouchDB listens only for connections from the local host. To change that edit /usr/local/etc/couchdb/couch.ini and restart CouchDB.
If you're running a firewall (you should be) then open the correct port.
sudo iptables -I INPUT 3 -p tcp --dport 5984 -j ACCEPTTesting that it all works
Since CouchDB talks HTTP we can use any HTTP client to check that it's running. Our web browser, for example. Fire it up and hit the IP address of the server on port 5984. If it's running and you can access it you should get back some details about the server.
{"couchdb":"Welcome","version":"0.8.0-incubating"}Love me!
If you've found this article useful I'd appreciate recommendations at Working With Rails.
Related articles
High Availability Apache on Ubuntu 8.04
It's nice when your website continues to be served even when something catastrophic happens. Running two Apache nodes and Heartbeat will help - if one server blows up, the other will take over in short order.
Prelude
You'll need two boxes and three IP addresses. I use virtual machines from Xeriom Networks. I've firewalled them and opened the HTTP port to the world.
sudo iptables -I INPUT 3 -p tcp --dport http -j ACCEPT
sudo sh -c "iptables-save -c > /etc/iptables.rules"For the purpose of this post, let's assume that the following IP addresses are available.
- 193.219.108.236 - Node 1 (craig-02.vm.xeriom.net)
- 193.219.108.237 - Node 2 (craig-03.vm.xeriom.net)
- 193.219.108.238 - Not assigned
Simple Service
First we'll setup Apache on both boxes. Nothing complex - we just want to make sure that we can serve something to HTTP clients.
Run the following command on both boxes.
sudo apt-get install apache2 --yesNow fire up a browser and hit the IP addresses assigned to Node 1 and Node 2. You should see the default Apache page stating "It works!". If you don't, check your firewall allows www traffic. Your firewall rules should look like the below - note the line ending tcp dpt:www.
sudo iptables -L
Chain INPUT (policy ACCEPT)
target prot opt source destination
ACCEPT all -- anywhere anywhere state RELATED,ESTABLISHED
ACCEPT tcp -- anywhere anywhere tcp dpt:ssh
ACCEPT tcp -- anywhere anywhere tcp dpt:www
DROP all -- anywhere anywhere
Chain FORWARD (policy ACCEPT)
target prot opt source destination
Chain OUTPUT (policy ACCEPT)
target prot opt source destinationAdding resilience
Apache can serve web pages from your machines now - that's great, but it doesn't protect against one of the machines dying. For that, we use a tool called heartbeat.
Install and configure Heartbeat on both boxes.
sudo apt-get install heartbeatNext we'll copy and customise the authkeys, ha.cf and haresources files from the sample documentation to the configuration directory.
sudo cp /usr/share/doc/heartbeat/authkeys /etc/ha.d/
sudo sh -c "zcat /usr/share/doc/heartbeat/ha.cf.gz > /etc/ha.d/ha.cf"
sudo sh -c "zcat /usr/share/doc/heartbeat/haresources.gz > /etc/ha.d/haresources"The authkeys should be readable only by root because it's going to contain a valuable password.
sudo chmod go-wrx /etc/ha.d/authkeysEdit /ec/ha.d/authkeys and add a password of your choice so that it looks like below.
auth 2
2 sha1 your-password-hereConfigure ha.cf according to your network. In this case the nodes are craig-02.vm.xeriom.net and craig-03.vm.xeriom.net. To figure out what your node names are run uname -n on each of the nodes. These must match the values you use in the node directives in the configuration file.
logfile /var/log/ha-log
logfacility local0
keepalive 2
deadtime 30
initdead 120
bcast eth0
udpport 694
auto_failback on
node craig-02.vm.xeriom.net
node craig-03.vm.xeriom.netWe need to tell Heartbeat we want it to look after Apache. Edit haresources and make it look like the following - still on both machines.
craig-02.vm.xeriom.net 193.219.108.238 apache2This file must be identical on both nodes - even the hostname, which should be the output of uname -n on node 1. The IP address should be the unassigned IP address given above in the prelude section.
In ha.cf we told Heartbeat to use UDP port 694 to communicate but because we're all nicely firewalled this port is blocked. Open it on both boxes.
sudo iptables -I INPUT 2 -p udp --dport 694 -j ACCEPTYour iptables rules should now look similar to the output below.
sudo iptables -L
Chain INPUT (policy ACCEPT)
target prot opt source destination
ACCEPT all -- anywhere anywhere state RELATED,ESTABLISHED
ACCEPT udp -- anywhere anywhere udp dpt:694
ACCEPT tcp -- anywhere anywhere tcp dpt:ssh
ACCEPT tcp -- anywhere anywhere tcp dpt:www
DROP all -- anywhere anywhere
Chain FORWARD (policy ACCEPT)
target prot opt source destination
Chain OUTPUT (policy ACCEPT)
target prot opt source destinationNow create a file on each box that tells us which webserver we're looking at.
# Node 1 (craig-02.vm.xeriom.net)
echo "craig-02.vm.xeriom.net" > /var/www/index.html# Node 2 (craig-03.vm.xeriom.net)
echo "craig-03.vm.xeriom.net" > /var/www/index.htmlCheck that this file shows up on each box by hitting the nodes IP addresses in the browser. If that works, it's time to flip the switch.
It lives... IT LIVES!
Start heartbeat on the master (node 1 / craig-02.vm.xeriom.net) then the slave (node 2 / craig-03.vm.xeriom.net).
sudo /etc/init.d/heartbeat startThis process takes quite a while to start up. tail -f /var/log/ha-log on both boxes to watch what's happening. After a while you should see node 1 say something like this.
heartbeat[6792]: 2008/06/24_11:06:21 info: Initial resource acquisition complete (T_RESOURCES(us))
IPaddr[6867]: 2008/06/24_11:06:22 INFO: Running OK
heartbeat[6832]: 2008/06/24_11:06:22 info: Local Resource acquisition completed.Testing for a broken heart
If you now check the output of ifconfig eth0:0 on both boxes you should see output like below.
# Node 1
sudo ifconfig eth0:0
eth0:0 Link encap:Ethernet HWaddr 00:16:3e:3c:70:25
inet addr:193.219.108.238 Bcast:193.219.108.255 Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1# Node 2
sudo ifconfig eth0:0
eth0:0 Link encap:Ethernet HWaddr 00:16:3e:92:ad:78
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1Node 1 has taken over our virtual IP address. If you kill Node 1, Node 2 will take it over. You can simulate this by taking down the Heartbeat process on Node 1.
# Node 1
sudo /etc/init.d/heartbeat stopChecking ifconfig again you should see that the virtual IP address has swapped nodes. If you bring up Node 1 again (start heartbeat) you should see the IP address swap back to that node.
If you got this far with no problems then congratulations, Heartbeat is running and your web tier will survive failure of a node. You can skip to the next section to see it working in the browser.
If you see some lines in the ha-log file telling you that the message queue is filling up then it's likely the two nodes can't communicate with each other. Check that you opened UDP port 694 on the firewall of both boxes.
heartbeat[6148]: 2008/06/24_11:05:09 ERROR: Message hist queue is filling up (500 messages in queue)Check the firewall rules look like below - the important line is the one ending in udp dpt:694.
sudo iptables -L
Chain INPUT (policy ACCEPT)
target prot opt source destination
ACCEPT all -- anywhere anywhere state RELATED,ESTABLISHED
ACCEPT udp -- anywhere anywhere udp dpt:694
ACCEPT tcp -- anywhere anywhere tcp dpt:ssh
ACCEPT tcp -- anywhere anywhere tcp dpt:www
DROP all -- anywhere anywhere
Chain FORWARD (policy ACCEPT)
target prot opt source destination
Chain OUTPUT (policy ACCEPT)
target prot opt source destinationThe proof is in the pudding
Mmm, cake.
Fire up your browser and hit the virtual IP address (193.219.18.238 in this post). You should see a page telling you that you're on Node 1.
Stop heartbeat (or shutdown Node 1) and hit the IP address again in the browser. You should now see that you're hitting Node 2.
Finally, bring Heartbeat back up on Node 1 (or start the box if you stopped it) and hit the IP address again. You should now be hitting Node 1 again.
Love me!
If you've found this article useful I'd appreciate beer and recommendations at Working With Rails.
Related articles
A simple email hub for your local network
I've been setting up the new Xeriom Networks MX service and decided that I'd document what I've done for your perusal. If you think something should be done in a different way, please do leave comments!
Requirements
The requirements for the MX service are pretty simple. We don't need to do spam filtering, blacklist checking, Greylisting, logging, virus scanning or anything like that. We're going to build a very simple service that provides reliable email delivery to hosts within our network and let our clients decide their own email policy.
Installing the software
I'll use Postfix because I'm pretty familiar with it. This is going to be pretty simple since we don't do any filtering; the basic Postfix install matches the requirements above.
sudo apt-get install postfix --yesStop Postfix here since it starts automatically after install.
sudo /etc/init.d/postfix stopConfiguring Postfix
Make /etc/postfix/main.cf specify the following values.
# Don't reveal the OS in the banner.
smtpd_banner = $myhostname ESMTP $mail_name
biff = no
# appending .domain is the MUA's job.
append_dot_mydomain = no
# Send "delivery delayed" emails after 4 hours.
delay_warning_time = 4h
readme_directory = no
smtpd_tls_cert_file=/etc/ssl/certs/ssl-cert-snakeoil.pem
smtpd_tls_key_file=/etc/ssl/private/ssl-cert-snakeoil.key
smtpd_use_tls=yes
smtpd_tls_session_cache_database = btree:${data_directory}/smtpd_scache
smtp_tls_session_cache_database = btree:${data_directory}/smtp_scache
# This is mx1.xeriom.net. Change for mx2, mx3, etc.
myhostname = mx1.xeriom.net
myorigin = mx1.xeriom.net
# Map root, abuse and postmaster to real email addresses.
virtual_alias_maps = hash:/etc/postfix/virtual
alias_maps = hash:/etc/aliases
alias_database = hash:/etc/aliases
mydestination =
relayhost =
mynetworks = 127.0.0.0/8
mailbox_size_limit = 0
recipient_delimiter = +
inet_interfaces = all
local_transport = error:No local mail delivery
local_recipient_maps =
smtpd_helo_required = yes
# Only allow the service to be used for hosts with final
# destinations within our VM network.
permit_mx_backup_networks = 193.219.108.0/24
# Only accept mail from nice people.
# Read and understand these blacklists policies before you
# use them or you risk losing mail!
smtpd_client_restrictions = reject_rbl_client zen.spamhaus.org,
reject_rbl_client cbl.abuseat.org,
reject_rbl_client dul.dnsbl.sorbs.net
# Only relay mail for which this machine is a listed MX backup.
smtpd_recipient_restrictions = permit_mx_backup, rejectCreate the aliases database and redirect abuse, root and postmaster mail to a real email address
newaliases
echo 'postmaster postmaster@xeriom.net' >> /etc/postfix/virtual
echo 'abuse abuse@xeriom.net' >> /etc/postfix/virtual
echo 'root root@xeriom.net' >> /etc/postfix/virtual
postmap /etc/postfix/virtualRestart Postfix so the changes take effect.
sudo /etc/init.d/postfix restartAfter installing, configuring and restarting the mail server we'll need to punch a hole in the firewall to allow traffic on the SMTP port. If you don't have a firewall set up, you should - set it up now.
sudo iptables -I INPUT 4 -p tcp --dport smtp -j ACCEPT
sudo sh -c "iptables-save -c > /etc/iptables.rules"Testing the setup
First, check that the new MX is listed in the zone and that the final MX is within the networks specified in permit_mx_backup_network. If they're not then edit the zone or the Postfix configuration. The domain that I'm testing this service with is emailmyfeeds.com.
dig MX emailmyfeeds.com +short
0 emailmyfeeds.com.
10 mx1.xeriom.net.
10 mx2.xeriom.net.
dig emailmyfeeds.com +short
193.219.108.60After doing that use telnet to send a trial email through the new MX box. Below is the entire SMTP conversation for a successful send.
telnet mx1.xeriom.net smtp
Trying 193.219.108.242...
Connected to 193.219.108.242.
Escape character is '^]'.
220 mx1.xeriom.net ESMTP Postfix
EHLO my-computer
250-mx1.xeriom.net
250-PIPELINING
250-SIZE 10240000
250-VRFY
250-ETRN
250-STARTTLS
250-ENHANCEDSTATUSCODES
250-8BITMIME
250 DSN
MAIL FROM: craig@xeriom.net
250 2.1.0 Ok
RCPT TO: craig@emailmyfeeds.com
250 2.1.5 Ok
DATA
354 End data with <CR><LF>.<CR><LF>
TEST!
.
250 2.0.0 Ok: queued as A6EED440BB
If, after you type the RCPT TO line you get an error something like 554 5.7.1 <test@foo.com>: Recipient address rejected: Access denied then the domain either doesn't have the MX currently listed in the zone file (or the change hasn't propagated through the DNS yet), or the final destination for the email doesn't fall within the ranges allowed by permit_mx_backup_networks.
You should also always, always check your MX's using an open relay checker - if you don't then you're helping spam distribution and I will hunt you down and hurt you.
Using the Xeriom MX service
If you're lucky enough to have a VM here at Xeriom Networks you'll be able to use this service from 2008-06-24 by following the instructions at http://wiki.xeriom.net/w/XeriomMXService.
