Telnet 101
Telnet has been around since before the dawn of (Unix) time but surprisingly few people know how to use this extremely useful tool to track down problems with many popular services. A few seconds with Telnet can save hours of frustrating searching, trial and error, and screaming at the computer.
Telnet can be used to simulate many plain-text protocols. I've used it to talk to MySQL, Memcached and Postfix. Here I'll show how to use it to test that an HTTP daemon can serve content over HTTP/1.1.
What is HTTP?
Before we can use telnet to simulate HTTP we need to know more about HTTP - what is it and how it works.
HTTP/1.1 - HyperText Transfer Protocol - is a plain-text protocol defined in RFC2616. It's used for a great many things, but the most visible one for most people is as a transport mechanism for web pages.
When you request a webpage a few special operations are performed (which I'll gloss over because they're beyond the scope of this article). After those have completed your web browser connects to the web server and sends it a requests for a page. A typical request for a web page looks something like this:
GET / HTTP/1.1
Host: example.com
The format of the request is:
[HTTPMETHOD] [PATH TO WEBPAGE] HTTP/1.1
Host: [SERVERNAME]
[BLANK LINE]The server responds with something similar except after the blank line there's usually the page you requested.
HTTP/1.1 200 OK
Server: Apache/2.2.3 (Red Hat)
Last-Modified: Tue, 15 Nov 2005 13:24:10 GMT
ETag: "b300b4-1b6-4059a80bfd280"
Accept-Ranges: bytes
Content-Type: text/html; charset=UTF-8
Connection: close
Date: Thu, 10 Dec 2009 10:37:33 GMT
Age: 7114
Content-Length: 438
<HTML>
<HEAD>
<TITLE>Example Web Page</TITLE>
</HEAD>
<body>
<p>You have reached this web page by typing "example.com",
"example.net",
or "example.org" into your web browser.</p>
<p>These domain names are reserved for use in documentation and are not available
for registration. See <a href="http://www.rfc-editor.org/rfc/rfc2606.txt">RFC
2606</a>, Section 3.</p>
</BODY>
</HTML>The format of the response is:
HTTP/1.1 [RESPONSE STATUS]
[RESPONSE HEADERS]
[BLANK LINE]
[REQUESTED WEBPAGE]There's a lot more to the HTTP protocol, but it's all document in rather a lot of (rather dry) detail in RFC2616. Mostly you can just skim the RFC for the parts you're interested in.
Simulated Awesome
Okay, so now we've got an idea how HTTP can be used to request a webpage. How do we go from that to using telnet to show it works?
The telnet manpage says that the telnet command can take arguments for host and port. In the above example we were requesting a page from example.com so that's the host we'll use. Typically HTTP is served on port 80, and this is true of example.com so we'll use that as the port.
telnet example.com 80With that we're connected to the HTTP service - the web server - on example.com. You'll see output something like this as connects:
Trying 192.0.32.10...
Connected to example.com.
Escape character is '^]'.Once connected you'll be left with your cursor on a blank line. This is where you can start pretending to be a web browser.
Type in the GET request shown in the "What is HTTP?" section above (including the blank line at the end). After a short delay you should get back the example.com webpage.
How is this useful?
Requesting a webpage by hand can quickly expose several common problems.
- Misconfigured firewalls mean telnet won't connect.
- The status code returned can tell you a lot about what the server did with your request. RFC2616 contains a list of HTTP status codes and their meaning.
- You avoid problems which may be caused by your browser caching the old version of files you request.
- You can easily check all the headers that the server sends back.
- By adding headers such as Accept-Encoding you can check that your assets are being served gzipped when it is requested.
Telnet isn't limited to checking just HTTP either. SMTP, IMAP, POP and many others can be simulated using telnet. It's not a silver bullet, but it's a very useful tool.
Simulating slow or laggy network connections in OS X
A client recently said that their site was loading slowly from remote sites. We got the specification of the network connection used, but I always forget how to do bandwidth limiting and latency simulation on OS X. This is a note for myself so I don't have to go searching again.
Configure a pipe that has the appropriate bandwidth limit and delay.
sudo ipfw pipe 1 config bw 16Kbit/s delay 350msAttach it to all traffic going to or from port 80.
sudo ipfw add 1 pipe 1 src-port 80
sudo ipfw add 2 pipe 1 dst-port 80Now traffic coming from or going to port 80 anywhere is limited by the pipe that you specified. Do your testing and once you get frustrated with slow access to the web remove the rules like so:
sudo ipfw delete 1
sudo ipfw delete 2Make sure you're @importing files that exist
A few people near me have heard the start of my rumblings about optimising the number of HTTP requests required for each page. There are a variety of reasons that you might want to do this, but that discussion is for another post. For now simply assume that I don't like unnecessary HTTP requests. Extrapolating from that you would reasonably assume that I really don't like wasted HTTP requests such as might happen when the url in a CSS @import directive 404's. If you assumed that, you'd be right.
I got fed up of tracking down the source of these errors in a few applications that I maintain so I fired up a TextMate session and scraped together this nasty little snippet of Ruby to do my dirty work for me.
#! /usr/bin/env ruby
css_root = File.expand_path(`pwd`.strip)
css_files = Dir[File.join(css_root, "**", "*.css")]
missing_imports = Hash.new([])
css_files.each_with_index do |css_file, index|
imports = File.read(css_file).split(/\n|\r/).grep(/\@import url\((.*)\)/)
imports.each do |import|
desired_path = import.scan(/url\((["'\ ])?(.*)\1\)/).to_a.first.to_a.last
desired_root = desired_path[0,1] == "/" ? css_root : File.dirname(css_file)
filesystem_path = File.expand_path(File.join(desired_root, desired_path))
if !File.exists?(filesystem_path)
missing_imports[css_file] += [{ :path => filesystem_path, :directive => import }]
end
end
end
if missing_imports.any?
puts "Missing files declared as imports in CSS:\n\n"
missing_imports.keys.each do |origin|
puts "Origin: #{origin}"
missing_imports[origin].each do |import|
puts "Missing @import file: #{import[:path]}"
puts "Directive: #{import[:directive]}"
end
puts ""
end
else
puts "No imported files are missing. Well done."
endRun it in the directory which serves your document root - for Rails applications this would be RAILS_ROOT/public/ - and it'll either spit out a list of missing files you've tried to @import or tell you that you've done well.
Missing files declared as imports in CSS:
Origin: /Users/craig/projects/1.8/public/stylesheets/.../find_by_service.css
Missing @import file: /Users/craig/projects/1.8/public/stylesheets/.../a_to_z.css
Directive: @import url('.../a_to_z.css');Now, just to be clear, I don't like @import directives. I'd prefer they were completely removed from the CSS. They are popular with a lot of people though so I'll compromise for the moment: if you must use them, please make sure they're going to work.
Using signals to debug long-running processes
Sometimes a long-running process will start to perform it's tasks much slower than it should, or in a strange order. It'd be lovely to know what the process is doing but strace produces a deluge of information that is several levels below what you want to know. What can you do?
Well, we could ask the process to turn on debugging output while it's still running. How does that sound?
trap("USR1") do
$DEBUG = !$DEBUG
@logger.level = $DEBUG ? Logger::DEBUG : Logger::INFO
@logger.info "USR1 received. Turning #{$DEBUG ? 'on' : 'off'} debugging."
endNow whenever you need more debugging information simple send a USR1 signal to your process.
kill -USR1 [pid]
