Log Output During Long-Running Processes

Long-running processes can appear to hang when you aren't given feedback. So let's give some feedback.

Often as a programmer I'm tasked with (or task myself with) writing ad hoc scripts or small projects for a one-time need. My most recent example was moving thousands of records from an old CMS to a new CMS, while maintaining associations and transforming the data in the process.

Tasks like data migration and transformations can get big fast and when running. They can appear to hang when really they are doing some heavy-lifting work behind the scenes. It can be difficult to know which is which if you don't provide yourself any output.

So, do it -- add logging to long-running processes. (Really they don't have to be long-running. You can add logging to any process.) Getting feedback is crucial to knowing what's happening within the bowels of your task-runner.

I often make my logging semantic. Say I'm migrating user data, maybe that feedback looks like this:

--- USER: Jenna Maroney ---
Exporting ... Done.
Transforming ... Done.
Importing ... Done.

--- USER: Mickey Rourke ---
Exporting ... Done.
Transforming ...

But on my most recent task there was enough data I didn't need to be printing out explicit information. I just needed to know if what I was doing worked or didn't work. So I used colored dots. If the process worked, I printed a green dot. If not, the dot was red. I concurrently collected the error in a temporary file that I could analyze after the script was done running.

Feedback during these processes can be super helpful. They let you know that something's actually happening. And then can also help you debug your code by figuring out where the issue is by looking at which message was the last to print to the console.

So add that feedback!

Let's Connect

Keep Reading

Parse a Web Page and Post to Slack Using Ruby

When another website is doing the heavy-lifting updating some content you want to keep up on, but there's no good integration with Slack, you can parse it yourself and post directly to a Slack channel.

Feb 24, 2016

The "Use With Love" Public License

Inspired by other public code licenses, this helped me better define exactly how I want to share my content and code.

Jun 18, 2021

Beware of Modern Web Lock-In

Although developer experience has brought much flexibility to modern web tools and services, vendor lock-in is still very much in play.

Sep 02, 2022