Choosing Among Log Management Tools

When you google log management tools, an interesting thing happens. At the time of this writing, you see no fewer than 4 paid ads, followed by a series of posts. These include, and this is not a joke, a post that lists the top 47.  As a software developer and tools consumer, this drives me insane. It probably does the same for you.

An author named Barry Schwartz coined a term (along with an eponymous book) for this frustration. He called it “the paradox of choice,” and it describes how, while we like to have some choice and autonomy, too much paralyzes us. To understand this in simple, terms, imagine selecting music for a dinner party. If offered two albums from which to choose, you’d make a pretty quick choice. If offered hundreds, you might thumb through them for a long time, trying to consider the likely tastes of all of your guests. And you might actually just give up eventually, and opt for only conversation with no background music at all.

The Paradox of Choice Among Log Management Tools

Back in the DevOps world, you face a similar plight when trying to pick among log management tools. You understand that you need a better way to aggregate and mine your logs than “by hand, using Sublime Text,” so you start to do some research. And then, about two searches in, you find yourself staring at post entitled, “The Top 47 Log Management Tools.” And, if you’re anything like me, you rub your temples and say to yourself, “ugh, never mind, I’ll figure this out tomorrow.”

That, of course, lines up with Schwartz’s findings about human behavior. Beyond having a few options, each additional option presented to a group of people causes fewer people to participate. The higher the number of log management tools in those posts, the fewer people will actually pick any of them at all.

Luckily, there’s a path back to joy. And it’s not even terribly complicated. You just need to dramatically narrow the field.

So today, I’m not going to add to the pile of “pros/cons/features” posts out there comparing dozens of tools. Instead, I’ll speak to heuristics you can employ to help you choose among log management tools. I’m going to help you narrow the field from a paralyzing number of choices that you make you unhappy to a manageable number that empowers you.

Read More

Five Reasons You Need Log Monitoring

You probably regard application logging the way you think of buying auto insurance. You sigh, do it, and hope you never need it. And aren’t you kind of required to do it anyway, or something? Not exactly the scintillating stuff that makes you jump out of bed in the morning.

It feels this way because of how we’ve historically used log files. You dutifully instrument database calls and controller route handlers with information about what’s going on. Maybe you do this by hand, or maybe you use a mature existing tool.  Or maybe you even use something fancy, like aspect-oriented programming (AOP). Whatever your decision, you probably make it early and then further information becomes rote and obligatory.  You forget about it.

At least, you forget about it until, weeks, months, or years later, something happens. Something in production blows up. Hopefully, it’s something innocuous and easily fixed, like your log file getting too big. But more likely some critical and maddeningly intractable production issue has cropped up. And there you sit, scrolling through screens filled with “called WriteEntry() at 2017-04-31 13:54:12,” hoping to pluck the needle of your issue from that haystack.

This represents the iconic use of the log file, dating back decades. And yet it’s an utterly missed opportunity. Your log file can be so much more than just an afterthought and a hail mary for addressing production defects. You just need the right tooling.

Log Monitoring To the Rescue

I’ve talked in the past about one form of upgrade from this logging paradigm: log aggregation. A log aggregation tool brings your log files into one central place, parses them, and allows you to search them rapidly. But you can do even more than that, making use of log monitoring via dashboards.

Read More

What Is Log Aggregation and How Does It Help You?

In order to understand the idea of log aggregation, you need to understand the pain it alleviates. You’ve almost certainly felt this pain, even if you don’t realize it.

Let’s consider a scenario that every programmer has probably experienced. You’re staring at some gigantic, dusty log file, engaging in what I like to think of as “programming archaeology.” And you have a headache.

A Tale of Logger Woe

It started innocently enough. A few users reported occasionally seeing junk text on the account settings screen. It’s not a regular bug, and it’s not particularly important. But it is embarrassing, and it seems like it should be easy enough to track down and fix. So you start trying to do just that.

You start by searching the database for the junk text in their screenshot. You find nothing.  Reasoning that application code must somehow have compiled the text in production, you figure you’ll head for the log files. When you open one up, and it crashes you text editor. Oops. Too big for that editor.

After using a little shell script magic to slice and dice the log file, you open it up and search for the text in question. That takes absolutely forever and yields no results. So you start searching for parts of the text, and eventually you have some luck. There’s a snippet of the text on line 429,012 and then another on line 431,114, with all sorts of indecipherable debug junk in between.

But you can’t find all of the text. And you have a headache. You then realize there’s a second log file for certain parts of the data access layer from before the Big Refactoring of ’15, and the rest of the text is probably in there. Your headache gets worse.

Read More