Using Signals to Debug Long-Running Processes

August 31, 2008 · 1 min read

Sometimes a long-running process starts performing its tasks much slower than it should, or in a strange order. You’d love to know what it’s doing, but strace produces a firehose of information several levels below what you actually care about. What can you do?

Well, you could ask the process to toggle its own debug output while it’s still running. Here’s how.

trap("USR1") do
  $DEBUG = !$DEBUG
  @logger.level = $DEBUG ? Logger::DEBUG : Logger::INFO
  @logger.info "USR1 received. Turning #{$DEBUG ? 'on' : 'off'} debugging."
end

Drop that into your process and, whenever you need more detail, just send it a USR1 signal:

kill -USR1 [pid]

Send it again to turn debugging back off. No restarts, no config file changes, no downtime. Just a clean toggle you can flip from the command line whenever curiosity strikes.

These posts are LLM-aided. Backbone, original writing, and structure by Craig. Research and editing by Craig + LLM. Proof-reading by Craig.