Java Exceptions and How to Log Them Securely

As a security consultant, I perform assessments across a wide variety of applications. Throughout the applications I’ve tested, I’ve found it’s common for them to suffer from some form of inadequate exception handling and logging. Logging and monitoring are often-overlooked areas, and due to increased threats against web applications, they’ve been added to the OWASP Top 10 as the new number ten issue, under the name “Insufficient Logging and Monitoring.”

So what’s the problem here? Well, let’s take a look.

Java Exceptions alert sign
Read More

Getting Started Quickly with Ruby Logging

Time for us to continue with our ongoing series, in which we teach you how to get started logging quickly in a variety of programming languages. We started out the series with C#, we proceeded to cover Java, and then we wrote about Python.

So, what about tipping the scale to the side of dynamically-typed interpreted languages? That’s exactly what we’re doing today by teaching you how to get up and running with logging, using the Ruby programming language.

Today’s post will follow the basic structure that’s been used in the previous articles. It will cover

  • How to implement a very rudimentary logger.
  • A discussion on the fundamentals of logging: why bother logging, which data to log, and where to log.
  • Finally, a very simple yet realistic example of proper logging, with help from the Ruby “Logger” class.

Like the previous installments of the series, we’ll create a very simple toy app in order to demonstrate how to log. As we’ve just said, we’re going to start with a very primitive—though functional—approach, and we’ll then evolve it toward a more sophisticated and realistic solution.

Ruby With Scalyr Colors

Read More

HTTP Monitor: What It Is and Why You Need It

One day, one of our main web APIs was down, and the first person that knew it was my boss. We were so worried about bringing the API up that we never paid attention to how he was able to be one step ahead of us. There were times when we even thought he had nothing else to do than constantly refresh the web page. But the truth is that he wasn’t doing that at all. He was using an HTTP monitor that emailed him every time the API was down, slow, or unresponsive.

It was actually lucky for us that he had that monitor: it helped everyone fix things before our clients could notice. But what is an HTTP monitor, anyway? And why else would you need it?


Illustration of Person Using HTTP Monitoring


Read More

Get Started Quickly With Python Logging

Picking up from the previous logging articles on how to get started logging with C# and Java, today we’ll be looking at how to get up and running quickly with logging in Python.

Even if you’ve already read the previous articles, this post is worth a read. It will cover new ground, like the basics around application logging in Python and a few other things, such as

  • Configuring the logging module.
  • What to log and why.
  • The security implications of logging.

So what are you waiting for? Keep reading, and let’s get a simple project set up to begin working with.

Python Scalyr Colors with LogRead More

A Detailed Introduction to the Apache Access Log

What is the Apache access log?  Well, at the broadest level, it’s a source of information about who is accessing your website and how.

But as you might expect, a lot more goes into it than just that.  After all, people visiting your website aren’t like guests at your wedding, politely signing a registry to record their presence.  They’ll visit for a whole host of reasons, stay for seconds or hours, and do all sorts of interesting and improbable things.  And some of them will passively (or even actively) thwart information capture.

So, the Apache access log has a bit of nuance to it.  And it’s also a little…complicated at first glance.

But don’t worry — demystifying it is the purpose of this post.

Apache Access Log: the Why

I remember starting my first blog years and years ago.  I paid for hosting and then installed a (much younger) version of WordPress on it.

For a while, I blogged into the void with nobody really paying attention.  Then I started to get some comments: a trickle at first, and then a flood.  I was excited until I realized that they were all suspiciously vague and often non-sequiturs.  “Super pro info site you have here, oPPS, I HITTED THE CAPSLOCK KEY.”  And these comments tended to link back to what I’ll gently say weren’t the finest sites the internet had to offer.

Yep.  Comment spam.

Somewhere between manually deleting these comments and eventually installing a WordPress plugin to help, I started to wonder where these comments were all coming from.  They all seemed to magically appear in the middle of the night and they were spammy, but I was interested in patterns beyond that.

This is a perfect use case for the Apache access log.  You can use it to examine a detailed log of who has been to your website.  The information about visitors can include their IP address, their browser, the actual HTTP request itself, the response, and plenty more.

An apache feather, representing our look at the apache access log.Read More

Get Started Quickly With Java Logging

You’ve already seen how to get started with C# logging as quickly as possible.  But what if you’re more of a Java guy or gal? Well, then we’ve got your back, too: today’s post will get you up to speed with logging using C#’s older cousin.

As in the previous post in this series, we’ll not only provide a quick guide but also go into more detail about logging, diving particularly into the what and why of logging.

The Simplest Possible Java Logging

For this simple demo, I’m going to use the free community version of IntelliJ IDEA. I’m also assuming that you have the Java JDK installed on your machine.

Read More

A Tale of Siri and My Home’s Energy Usage

Full disclosure: I’m a Scalyr DevOps engineer, but I’d be geeking out over the sheer number of possible uses for Scalyr even if I wasn’t. It’s more than a log analysis tool—it’s a platform. Scalyr now monitors the temperature inside my house, as well as the history of my thermostat and HVAC system usage. I’m one of very few homeowners in the world with real-time access to information about my HVAC system’s energy usage.

What compelled me to do this? Siri.

And a desire to harness home automation to improve my house’s energy efficiency. Here’s the story.

Read More

Irreversible Failures: Lessons from the DynamoDB Outage

Summary: Most server problems, once identified, can be quickly solved with a simple compensating action—for instance, rolling back the bad code you just pushed. The worst outages are those where reversing the cause doesn’t undo the effect. Fortunately, this type of issue usually generates some visible markers before developing into a crisis. In this post, I’ll talk about how you can avoid a lot of operational grief by watching for those markers.
Read More

99.99% uptime on a 9-to-5 schedule

Running a 24/7 Log Monitoring Service

Being “on call” is often the most dreaded part of server operations. In the immortal words of Devops Borat, “Devops is intersection of lover of cloud and hater of wake up at 3 in morning.” Building and operating sophisticated systems is often a lot of fun, but it comes with a dark side: being jarred out of a sound sleep by the news that your site is down — often in some new and mysterious way. Keeping your servers stable around the clock often clashes with a sane work schedule.

At Scalyr, we work hard to combat this. Our product is a server monitoring and log analysis service. It’s internally complex, running on about 20 servers, with mostly custom-built software. But in the last 12 months, with little after-hours attention, we’ve had less than one hour of downtime. There were only 11 pager incidents before 9:00 AM / after 5:00 PM, and most were quickly identifiable as false alarms, dismissible in less time than it would take for dinner to get cold.

In this article, I explain how we keep things running on a mostly 9-to-5 schedule.

Read More

Impossible Engineering Problems Often Aren’t

When your problem is impossible, redefine the problem.

In an earlier article, I described how Scalyr searches logs at tens of gigabytes per second using brute force. This works great for its intended purpose: enabling exploratory analysis of all your logs in realtime. However, we realized early on that some features of Scalyr―such as custom dashboards built on data parsed from server logs―would require searching terabytes per second. Gulp!

In this article, I’ll describe how we solved the problem, using two helpful principles for systems design:

  • Common user actions must lead to simple server actions. Infrequent user actions can lead to complex server actions.
  • Find a data structure that makes your key operation simple. Then design your system around that data structure.

Often, a seemingly impossible challenge becomes tractable if you can reframe it. These principles can help you find an appropriate reframing for systems engineering problems.

Read More