What Goes Into Log Analysis?

I’ve talked here before about log management in some detail.  And I’ve talked about log analysis in high-level terms when making the case for its ROI.  But I haven’t gone into a ton of detail about log analysis.  Let’s do that today.

At the surface level, this might seem a little indulgent.  What’s so hard?  You take a log file and you analyze it, right?

Well, sure, but what does that mean, exactly?  Do you, as a human, SSH into some server, open a gigantic server log file, and start thumbing through it like a newspaper?  If I had to guess, I’d say probably not.  It’s going to be some interleaving of tooling, human intelligence, and heuristics.  So let’s get a little more specific about what that looks like, exactly.

Log Analysis, In the Broadest Terms

In the rest of this post, I’ll explain some of the most important elements of log analysis.  But, before I do that, I want to give you a very broad working definition.

Log analysis is the process of turning your log files into data and then making intelligent decisions based on that data.

It sounds simple in principle.  But it’s pretty involved in practice.  Your production operations generate all sorts of logs: server logs, OS logs, application logs, etc.  You need to take these things, gather them up, treat them as data, and make sense of them somehow.  And it doesn’t help matters any that log files have some of the most unstructured and noisy data imaginable in them.

So log analysis takes you from “unstructured and noisy” to “ready to make good decisions.”  Let’s see how that happens.

 

Read More

Search Your Files with Grep and Regex

How do you search through a file?  On the surface, this might seem like sort of a silly question.  But somewhere between the common-sense answer for many (“double click it and start reading!”) and the heavily technical (“command line text grep regex”) lies an interesting set of questions.

  • Where does this file reside?
  • What kind of file is it?
  • How big is the file?
  • What, exactly, are you looking for in the file?

Today, we’re going to look at one of the most versatile ways to search a file: using grep and regex (short for regular expression).  Using this combination of tools, you can search files of any sort and size.  You can also search with extremely limited access to your environment, and if you get creative, you can find just about anything.

But with that versatility comes a bit of a learning curve.  So let’s look at how to take the edge off of that and get you familiar with this file search technique.  To do that, I’ll walk through a hypothetical example of trying to extract some information.  First, though, let’s cover a bit of background.

Magnifying Glass Aimed at Search Box on Screen

Read More

Logging Levels: What They Are and How They Help You

Logging levels probably aren’t the most exciting thing in this world.  But then again, neither is banking.  And yet both things are fundamental to the people who use them as a tool.

Application logging is one of the most important things you can do in your code when it comes to facilitating production support.  Your log files serve as a sort of archaeological record of what on earth your codebase did in production.  Each entry in a log file has important information, including a time stamp, contextual information, and a message.  Oh—and generally, something called a logging level.

So what are logging levels?

Well, put as simply as possible, they’re simply a means of categorizing the entries in your log file.  But they categorize in a very specific way—by urgency.  At a glance, the logging level lets you separate the following kinds of information:

  • Hey, someone might find this interesting: we just got our fourth user named Bill.
  • OH NO SOMEONE GET A FIRE EXTINGUISHER SERIOUSLY RIGHT NOW.

Logging levels can help you distinguish whether you need the pictured fire extinguisher or whether the message just contains information.

For the most part, this distinction helps in two ways.  First, you can filter your log files this way during search.  And second, you can control the amount of information that you log.  But we’ll get to that in a bit.

Logging Levels: Why Do We Do It?

When it comes to logging, you have two essential and opposing forces, if you will.  On the one hand, you want to capture every last detail you can because this might prove useful during troubleshooting or auditing your system.  But on the other hand, all of that logging consumes resources. You can eat up disk space, overload people reading the logs, and even start to slow down your production code if you go overboard.

So logging requires either a balance or a way to get both the proverbial signal and the noise.  And logging levels look to help with this by making your situation configurable on the fly.

Here’s a helpful metaphor, perhaps.  Imagine you have an android friend and you ask him about what he did today.

“Well, I woke up, inhaled, moved my nose three inches to the right—”

“No!  Too much information.  Go from granularity 10 down to 3.”

“I woke up, had breakfast, caught a taxi…”

“THAT’s better.”

Usually, you don’t need to know everything the android did.  But if he starts malfunctioning, you might want to go back to “granularity 10” to zoom in on the problem as narrowly as possible.

Logging levels work this way.  Often you’ll only want to know if there are problems or warning conditions.  But sometimes you’ll really want to dial up the information for troubleshooting purposes.

Read More

Get Started Quickly With C# Logging

If you’re interested in getting started with C# logging as quickly as humanly possible, you’ve come to the right place.  Today, we’ll look at just how to do that.  Don’t worry, we’ll get into comprehensive detail too, covering a number of things.  Some of those include

  • What is logging?
  • Why would you want to log?

But for now, let’s get to the action.  How can you start logging in the absolute quickest and simplest way possible?  Let’s take a look.

Getting started quickly with C# logging

The Simplest C# Logging That Could Possibly Work

Let’s start with a clean slate for the sake of easy demonstration.  First, open Visual Studio and create a new console project by selecting File->New->Project, as shown here.

To get started with C# logging, create a console project, as shown here.

I’ve called my new project ConsoleProject and opted to put it into a new solution called LoggerPlaypen.  No need to congratulate me on my creativity, though.

This will open the Program.cs file containing your Program class.  Let’s now add a few lines of code to that class, as follows:

static void Main(string[] args)
{
    var displayMessage = "Hello, user!";

    Console.WriteLine(displayMessage);
    Console.ReadLine();
}

We now have an application that successfully defines a greeting, displays it, and waits for an enter keystroke to dismiss it.  When you hit F5 to run, you’ll see the following message before pressing enter:

Here is the console output, "Hello, user!"

But let’s say that we wanted not only to display this message but also to log it to a file.  How might we do that?  With something fancy, like a logging framework, inversion of control, or aspect-oriented programming?  Nah, let’s not get ahead of ourselves.  Try this instead, making sure to add “using System.IO;” at the top with the other using statements:

static void Main(string[] args)
{
    var displayMessage = "Hello, user!";

    File.WriteAllText("log.txt", displayMessage);

    Console.WriteLine(displayMessage);
    Console.ReadLine();
}

Now, when you hit F5, you’ll see your greeting message once again.  But that extra line you’ve added will create a log file for you.  Go to your project directory, and then navigate to the Bin->Debug folder. You’ll see a file hanging out there in addition to your console executable.

Here's the file in your Debug directory, plus a shot of the file's contents, "Hello, user!"

Is That Really It?

And…that’s it.  You just successfully implemented your first C# logger!

Is that all you need to know about logging?  Of course not.  Will this logging scheme be sufficient for any but the most trivial applications?  Nope.  But have you learned the barest essential backbone of C# logging?  Absolutely.

I told you it’d be a quick start.  But I strongly recommend that you read on to really get a handle on this logging thing.

Read More

Surprising Use Cases for Log Visualization

People commonly say that a picture is worth a thousand words.  So I wonder if log visualization is worth a thousand log entries.  The math equivalency might be a little hard to prove, but the idea is worth exploring.

You’re recording all sorts of information in your log files, but are you visualizing that information?

Do you have dashboards and graphs that help you picture production behavior?  Or does the information sit buried within digital mountains of arcane strings?  The proverbial needle in the haystack?  Does anyone who wants to use it need to engage in laborious searches?

If you’re not visualizing your logs, you’re missing out.  But I don’t necessarily want to make the case for visualization today.  Instead, I’d like to offer some ideas for visualization that you perhaps hadn’t considered.  Let’s take a look at some use cases for log visualization that you might not have considered.

Prerequisites for Log Visualization

First things first, though.  Before I can take you through the use cases, you need to have a setup that allows log visualization.  Specifically, you need modern log file management, which includes the following things that concern us:

  • Log aggregation (gathering and storing log files).
  • Meaningful parsing of the data contained in the log files.
  • Powerful search capabilities.
  • Log visualization tools.

Don’t get me wrong.  You could implement all of this stuff yourself.  But then again, you could also implement your own source control and text editor.  Just because you can doesn’t mean you should.

If you want to get serious about visualizing log files without burning tons of time, you need tooling and infrastructure in place to help you.  Once you have that, though, let’s take a look at some of the things you might do.

Read More

Creating an Audit Trail for Your Business

No matter what you do, there will be aspects of your job that you absolutely love.  And then you’ll have the things that you tolerate out of necessity.  I’m guessing that, for almost everyone reading, “audit trail” sounds like something that fits squarely into the “tolerate” bucket.

Even if you don’t know what it is, it probably sounds equal parts intimidating and boring.  The closest word association you’ll likely have with “audit” is that it’s what the IRS does to you when it simultaneously takes a fine-toothed comb to your life and demands more money from you.  And looking to avoid angering the IRS is probably not what you dreamed of on career day as a child.

But building and maintaining an audit trail for your business doesn’t have to be onerous.  Far from it.

Magnifying glass looking at a graph

What Is an Audit Trail, Anyway?

I’ve thrown the word around a few times, but let’s get a little more precise to set the stage for a post.  What is an audit trail?

To get a good working definition of “audit trail,” consider the definition of “audit.”

An official examination and verification of accounts and records, especially of financial accounts.

It has official overtones to it, and it involves taking a detailed look at relevant records.  So when you commission an audit, you ask someone to come in, on the record, and take a detailed look at what you’re doing.

Building and maintaining an audit trail for your business doesn't have to be onerous.

An audit trail, then, is what you do to facilitate this activity.  You make sure to dutifully document and capture anything that an auditor might need.  What’s the reasoning for this?  Generally speaking, you do this to demonstrate that you operate with a high degree of transparency and that your activities are all ethical, responsible, and legal.

Take the aforementioned case of the IRS mandating an audit for you.  This will tend to go much better for you if you’ve made sure to create an audit trail: saving receipts, noting business expenses, keeping careful track of all income, etc.

 

Read More

Building a Sustainable Startup

Although Scalyr has been around since 2011, it feels like we are really just getting started.

So many tech startups come roaring out of the gate, pursuing growth at all costs. Sometimes this leads to spectacular success, but much more often it leads to burnout and retrenchment, if not outright failure. Many promising startups have failed due to early over-reach.

At Scalyr, we’re taking a different approach. We spent over three years with a small team, literally above my garage, taking our time to build the right product in the right way. Only after we’d built a differentiated product that our early users loved did we set out to grow the team and the business.

We have been absolutely blown away by the results:

  • Customer devotion: since we signed our first customer in mid-2013, we have not had a customer leave us for another solution. The sheer performance of our log management service has been a pronounced and sustained differentiator.
  • Word of mouth: once adopted, Scalyr tends to spread within an organization. We have had multiple instances of customers beginning at five-figure annual revenue and growing to seven figures.
  • Scalability: through multiple orders of magnitude of growth, we’ve been able to maintain the performance and functionality that makes Scalyr special.

We are building a real business with real customers, and I’m excited to share some of our recent progress.  

 

Origin story

Rewind to 2006. Amazon EC2 and S3 were still under wraps, Facebook was only available to .edu addresses… and Google had just acquired my startup, Writely – soon to be known as Google Docs. Pretty soon, I was leading a project to build a new storage infrastructure for applications such as 

Docs, Sheets, Drive, and Picasa. Google has a strong culture of internal tool development, and we soon found ourselves using 17 different operational visibility tools to maintain a reliable service. Seventeen! Together, they provided a lot of functionality, but juggling that many tools was a bit of a nightmare.

It was clear that there was a lot of room for improvement. Around the industry, it’s more common to see teams using four or five visibility tools, rather than 17. But everyone suffers from too many tools, too little insight, and too much time spent investigating issues. In 2011, after leaving Google, I started Scalyr to create a better solution. Our ultimate goal is to revolutionize operational visibility, making it easy to understand the behavior of modern, complex cloud stacks.

The first step on this journey is our log management service. Logs provide the most detailed view of server and application behavior and are a critical piece of the operational puzzle. But existing log management tools were so clunky and slow that people avoided using them. Most of these tools are built on traditional keyword indexes, a technology originally designed to search books. Rethinking the problem from first principles, we built a profoundly more efficient solution, proving blazing-fast search over terabyte-scale aggregated logs.

The early response was beyond what any of us thought possible. We raised $2M in seed funding in 2015 to jumpstart the business.

 

Building a Sustainable Business

We have a long way to go to fulfill our long-term vision. To accomplish that, we have to build a sustainable business with solid fundamentals. We’ve been thoughtful about our growth to date, building the team in proportion to revenue. In fact, we’ve been hovering around breakeven – some months even profitable – for the past several quarters.

Sustainable growth requires a delicate balance: be aggressive enough to seize opportunities, but conservative enough to maintain company culture and healthy finances. Combining growth with healthy, sustainable practices requires more than simply pacing yourself. It requires efficiency. With VC cash burning a hole in your pocket, it’s tempting to throw money at all your problems; but then you’re quickly looking for your next round, and you’ve fallen off the sustainable path. So we’re always looking for ways to become more efficient.

It helps tremendously that we have a sustainable product differentiation – performance – that’s linked to a fundamental technological advantage.

 

Powering the next stage

With solid revenue, delighted customers, and a clear path forward, we’re limited only by the speed at which we can execute. So I’m excited to share that we have raised a $20M Series A led by Shasta Ventures, with participation from Heroic Ventures, Susa Ventures, and Bloomberg Beta – bringing our total amount raised to $28M. 

 

By the way, we’re hiring 🙂

This is the 6th startup I’ve [co]founded. People sometimes ask which one is my favorite. Writely was certainly the splashiest. Spectre was pretty cool (and was the last time I got to hack assembly language). But I can say without hesitation, Scalyr is the most satisfying, rewarding project I’ve been lucky enough to work on. I truly believe this is going to be one of those companies where people will look back and say “I wish I’d been there when…” – well, right now is “when”.

There couldn’t be a more exciting time to join! We’re hiring across all teams – especially engineering, sales, marketing, and customer success. Take a look at our Careers Page and become part of the journey.

Log File Too Big — What Should I Do?

You have a problem. But you don’t just have an ordinary problem. You have one of the most frustrating kinds of problems in the technical world. In the most basic terms, you’re trying to open a log file that’s too big to open. But “log file too big” doesn’t fully capture the frustration or the problem.

You need something out of your log file, so you go to open it. Then you wait. And wait. And wait.

After some amount of time, your text editor just crashes. Hoping it’s a fluke, you try again, waiting 15 minutes before another crash. So you’re half an hour in and not only have you not solved your actual problem — you haven’t even successfully taken what should be the simplest imaginable step toward solving it.

This combination of a long feedback loop with a non-deterministic outcome is what makes this so maddeningly frustrating. But fear not. Let’s take a look at how you can solve this, starting with the quickest and most superficial route and working toward root cause analysis.Log file too big? It can seem as though the thing is crushing you.

Pick a Different Tool at Your Disposal

If opening this log file is crashing an editor, like, say, Notepad, then your easiest step is to use a different editor. At least that way you can know that fate will reward your waiting with an opened file rather than with a crash.

Your path of least resistance here is to use something you already have installed. So consider the following utilities for each application.

  • For Windows, you can use WordPad. If you have enough memory to cover the size of the file you want to edit, WordPad will load it. So these days, that’s quite likely to apply to files even topping a gig in size.
  • For Mac, use Vim. It should be able to handle as big a file as you have memory, and with good search besides.
  • There are a lot of different flavors of Linux out there, so it’s a little harder to talk about default installations. But if you have it, you can also use Vim here. If not, you can install it easily, and you can use tail -X at the command line, where X is the number of lines you’d like to see.

That should at least get you started. You should be able to see your file without needing to wait for something to maybe or maybe not crash.

Download and Use a Text Editor Meant for This

If you have a little more patience, you should ask yourself whether your current need is a one-time-only situation or if you’ll be viewing and editing a lot of large files. If the latter, you’ll want to get more deliberate about the tools in your toolbox. I might suggest this even if you think this is a one-time need. Familiarizing yourself with a new, powerful text editor can’t hurt anything.

The number of text editors available to you is FAR too large to enumerate here. But Wikipedia has an extensive page on them, including specific information about file size.

If the problem solved by opening your large file isn’t too pressing, you could always engage in some yak-shaving. But you should probably solve that problem first, using a tool at your disposal. Then come back and spend some time evaluating your text editor options. Find one that can open large files and that has other features you like besides. I’d say even try out a few of them.

If large files figure to be part of your life going forward, you should have a plan of attack for them.

Read More

How You Can Shorten the Defect Life Cycle

Ah, the software defect. It’s the bane of our collective existence, and it also seems unavoidable. Okay, frankly, it probably is unavoidable, for all intents and purposes. But that doesn’t mean we’re powerless to do anything about it. We can chip away at its impact by reducing its severity and by shortening the defect life cycle.

What Is the Defect Life Cycle?

Muting the impact of defects is a self-explanatory endeavor, but what do I mean by “defect life cycle”? Well, first, consider the word “cycle.” This borrows from the lean idea of cycle time, which is fancy Six-Sigma speak for “How long does it take from start to finish?”

Okay, so why not just call it “defect lifetime”? I suppose I could have used that term. But it omits a subtle yet crucial consideration. A defect in our software moves through a series of phases and steps as people work to correct it.

“Lifetime” makes it sound like fate gives birth to the defect, and then it simply exists until it naturally expires. But that’s not at all what happens. Instead, the team collaborates methodically to track down the defect, assess it, address it, and roll out a fix of some kind. So we think about a defect life cycle rather than a defect just living its life quietly out in the country somewhere.

But terminology and philosophy aside, how do you shorten the defect life cycle?

Production defects tend to generate stress and keep people up at night. From the moment a user reports it until the moment someone resolves the problem, tensions run higher. Let’s take a look at how to reduce the length of that tense time.

Read More

Log Management: What Is It and Why You Need It

To understand log management, you first need to understand what problem it solves. Once you see that, you’ll know both what it is and why you need it.

Software these days involves a lot of complexity that didn’t exist once upon a time. We’ve moved things into the cloud, created software/platforms/infrastructure as services, and embraced distributed computing.

That’s a sea change from the good ol’ days of the 1990s. Back then, you’d write a bunch of code, build it, put it on CDs or floppy disks, and mail it to people. It’s even a sea change from the 2000s, when the web application took over. Instead of CDs, you’d set up a web server, deploy your software to that, and let users and their browsers have at it.

But today, we have containers and microservices. We have software intelligence distributed around the globe, spinning up and down on demand, collaborating and orchestrating. We’ve traded the simplicity of the historical monolith for the flexibility and complexity of distributed intelligence.

Log Files in a Distributed World

Think about the change I’ve just described. And now imagine what that means for the existence of a log file.

In the 1990s, you’d add code to your application that dumped information to a single log file. If your users had problems, they could zip up that log file, along with an OS log file for good measure, and send those to you for troubleshooting. With 2000s web applications, that same application log file, along with the web server log file and the database log file, did the trick.

But now? Good luck. Your production operations include six RESTful microservices on six different servers, a bunch of on-demand containers, a few miscellaneous web apps, a service bus, and who knows what else? Each of those concerns is contained, isolated, simple, and useful.

But troubleshooting across those concerns, when the issue happens in the gaps, can be a mess. And gathering 20 different log files that you attempt to reassemble into some facsimile of order doesn’t help matters at all.

Log Management to the Rescue

That is where the idea of log management as a first class need enters the picture. If you have a desktop app or a simple web app, you can probably get by with grep, text editors, and elbow grease. But as soon as you grow beyond that, you’re going to need a better approach.

Log management is that better approach. Instead of regarding your applications’ logs as separate, unrelated entities, you conceive of them as parts of a whole. You weave them together and then use them to paint a dynamic, intelligent, and visual picture of the health of all your systems.

If that sounds daunting, don’t worry. You don’t need to implement all of this yourself. In fact, you definitely shouldn’t do it yourself any more than you should write your own source control. A lot of talented toolmakers have invested significant effort in helping you with your log management.

But rather than focus on specific tools, let’s take a look at log management as a function of its components. What does a good log management scheme involve, and what should you expect out of it?

Read More