Scalyr Platform: Batch Log Export, Alerting, and UI

In the Scalyr platform, releases over the past month include batch log export to Amazon S3 and alerting and UI improvements.

Batch Log Export to Amazon S3

We will make batch log export available on June 11, allowing you to filter logs and export them to Amazon S3. This lets you archive log data for compliance purposes, store them for future use, or include them in your CRM or ticketing system. Moreover, the exports run in the background in Scalyr. Users receive email notifications when they are completed. You can read more at Batch Export to Amazon S3.

Scalyr Alert Email
Scalyr Alert Email

Alerting Improvements

As part of our effort to improve alerting, we’re making email notifications easier to use. We have grouped alerts into “new,” “ongoing,” and “resolved” sections, making new issues stand out more clearly. Additionally, each alert now provides direct, deep links to the exact spot in Scalyr search or graphs for the alert time range (or the current time).

User Interface Improvements

We have upgraded the UI, including adding expandable dashboard legends so you can make better sense of what you’re seeing.

Scalyr Dashboard Expandable Legend
Scalyr Dashboard Expandable Legend

Going Forward

We are developing Scalyr for the engineering front line with a focus on our three value pillars: fast, simple, and shareable.

We will continue to revamp common workflows to refine the user experience. Right now, we are focusing on alerts. Soon, we will add improved alert management, including bulk actions and better silencing options, plus a more useful alert landing page.

Feedback

Your product (or any) feedback is always welcome. Please reach out to us at support@scalyr.com.

Getting Started Quickly With Go Logging

It’s time to talk about how to get started with logging again. The languages we’ve covered so far are C#, Java, Python, Ruby, Node.js, and JavaScript. Today, we’re going to be talking about the Go programming language, also known as Golang.

Go is a statically compiled, open-source programming language created by Google in 2009. It runs on Windows, Linux, and Mac and has been increasingly adopted in the last couple of years because of its simplicity and concurrency mechanisms. And in case you didn’t know, Docker and Kubernetes were written in Go.

Go is quite an interesting language when we talk about logging. Why? For one thing, it includes a really simple native library for logging. But the most interesting thing is that in Go there are no exceptions—only errors. That means when you want to log application errors, things get interesting. We’ll cover that later in this post.

But let’s start by taking a look at how easily you can get started logging in Go.

Go Beaver With Scalyr Colors

Read More

Log Analyzer: What It Is and How It Can Help You

A log analyzer allows you to quickly search a specific set of data in a haystack of records. Think of it like bookkeeping. Accounting books tell a story for each transaction made in a period of time. To search through those transactions, you need criteria like the date and amount paid. You find the transaction log and it has more information: payment method, recipient, plus other details.

That’s what a log analyzer does: it helps you find events in a collection so you can study them and fix problems. Here’s an ideal case:

  1. An incident report provides data for log analysis.
  2. Someone reproduces this issue based on your log data.
  3. Your development team proposes a solution in a new fixed app.
  4. Someone reproduces the incident again, this time using your new app.
  5. Log analysis and testing show whether this fix was effective or not.

In this article, I’ll share insights from my experience with log analysis. Read on, draw parallels with your organization, and infer your own conclusions.

Graph with a magnifying glass over it

Read More

The 10 Commandments of Logging

When you search for things on the internet, sometimes you find treasures like this post on logging, e.g. creating meaningful logs.

This post, from a few years back, is authored by Brice Figureau (found on Twitter as @_masterzen_). His blog clearly shows he understand the multiple aspects of DevOps and is worth a visit.

In recent discussions with Brice, he stated that he believes the rules are still quite valid with some tweaks needed for containers and Kubernetes and that in today’s world of GDPR, logging sensitive data is a non -starter. But these rules are informative and worth consideration.

Our thanks to Brice for letting us repost his blog under Creative Commons CC-BY.

Guest author Brice Figureau

2 stone tablets with Roman numerals from 1 to 10
Copyright: moises / 123RF Stock Photo

After writing an answer to a thread regarding monitoring and log monitoring on the Paris DevOps mailing list, I thought back about a blog post project I had in mind for a long time.

I wrote this blog post while wearing my Ops hat and this is mostly addressed to developers.

Knowing how and what to log is, to me, one of the hardest tasks a software engineer will have to solve. Mostly because this task is akin to divination. It’s very hard to know what information you’ll need during troubleshooting… That’s the reason I hope those 10 commandments will help you enhance your application logging for the great benefits of the ops engineers 🙂

1. Thou shalt not write log by yourself

Read More

CI/CD Tools: How to Choose Yours Wisely

Continous integration (CI) and continuous deployment (CD) tools allow teams to merge, build, test, and deploy code branches automatically. Implementing them along with conventions like “commit frequently” forces developers to test their code when it’s combined with other people’s work. Results include shorter development cycles and better visibility of code evolution among different teams.

Once you commit to using CI/CD in your software development cycle, you’re immediately faced with a galore of options: Travis, Jenkins, GitLab, CodeShip, TeamCity, and CircleCI, among others. Their names are catchy, but they hardly describe what the tools do. So here’s a roadmap for choosing the right tool for your needs.

CI/CD Tools: Choosing Wisely

Read More

Five To-Dos When Monitoring Your Kubernetes Environment

If you’re on the DevOps front line, Kubernetes is fast becoming an essential element of your production cloud environment. Since container orchestration is critical to deploying, scaling, and managing your containerized applications, monitoring Kubernetes needs to be a big part of your monitoring strategy.

Container environments don’t operate like traditional ones. So, if you are monitoring your applications and infrastructure, you need to be thoughtful about how you monitor your container environment in which they are running. Here are five best practices to inform your strategy:

  1. Centralize your logs and metrics. Orchestrating your containerized services and workloads through Kubernetes brings order to the chaos, but remember that your environment is still decentralized. You will give yourself a fighting chance if you centralize your logs and metrics.
  2. Account for ephemeral containers. The beauty of container orchestration is it’s easy to start, stop, kill, and clean up your containers in short order. However, monitoring them may not be so easy. You still need to debug problems and monitor cluster activity, even when services are coming and going. The trick is to grab the logs and metrics before they’re gone. If you don’t, your metrics will look more like the graph on the left than the one on the right.
    log files examples for transient containers
  3. Simplify, simplify, simplify. With all of the moving pieces in your container environment (services, APIs, containers, orchestration tool), you need to monitor without introducing unneeded complexity. Rather than bloating your container with various monitoring agents, each requiring updates on unique schedules, abstract your monitoring and management tools from what you’re monitoring and managing. This will also help your engineers focus on building and delivering software, not operating the delivery platform.
  4. Monitor each layer explicitly. You will need to collect logs and monitor for errors, failures, and performance issues at each layer – the pod, the container, and the controller manager – of your environment. For example, you’ll need to be able to troubleshoot pod issues, ensure the container is working, and collect runtime metrics in the controller manager.
  5. Ensure data consistency across layers. For fast, accurate debugging, you need to ensure data consistency across all the layers in your container environment. Things like accurate timestamps, consistent units of measurement (such as milliseconds vs. seconds), and collecting a common set of metrics and logs across applications and components will help you troubleshoot and debug quickly and accurately across all of your layers.

One best practice for accomplishing these to-dos in a simple, straightforward manner is to monitor the containers in your Kubernetes environment without touching your application containers. Do this by introducing a DaemonSet, or alternatively a sidecar, into your Kubernetes environment(s) that sits alongside your containerized services and includes your logging and metrics collection agent. Deploying in this method will ensure consistent data collection, minimize the changes required to your application containers, and most importantly, eliminate the possibility of selective blindness in your production environment.

A few ways to implement this include:

  • Introduce a DaemonSet with the Fluentd logging agent (this will give you logging but not metrics). If you already have an ELK cluster configured, this is probably the option for you. Learn more here.
  • Introduce a DaemonSet or sidecar with the Prometheus metrics agent (CoreOS has done an excellent job of integrating Prometheus and Kubernetes). Running Prometheus on your Kubernetes cluster will give you metrics instrumentation, querying, and alerting. Learn more here.
  • A variety of metrics and performance monitoring tools, including Heapster, DataDog, cAdvisor, New Relic, Weave/VMware, and several others also offer a DaemonSet or sidecar options for Kubernetes monitoring.
  • Scalyr, log management for the DevOps front line, has a preconfigured DaemonSet containing the open source Scalyr agent available for download and use. The Scalyr DaemonSet natively supports both Kubernetes logging and metrics. You can download the YAML file for deploying the containerized Scalyr agent from GitHub here. Note that you also can download the full open-source Scalyr agent from GitHub here.

 

Network Traffic Monitoring: The 7 Best Tools Available To You

Sales, pre-sales, human resources, the company cafeteria: they’re all online. If the network is down, employees are angry and customers have gone elsewhere. That’s why network traffic monitoring is a critical part of maintaining a healthy enterprise.

Fixing network problems when they happen isn’t good enough. IT managers have to proactively watch systems and head off potential issues before they occur. This means observing network traffic and measuring utilization, availability, and performance.

A useful monitoring tool offers these features:

  • real-time network monitoring
  • an ability to detect outages in real time
  • a mechanism for sending alerts
  • integrations for network hardware, such as SNMP and NetFlow monitoring

This is a list of the best tools available for monitoring your network traffic. Several of them are sold as SaaS, others for running on-premises, and a couple are open-source with optional commercial versions. All of these tools offer more than just network monitoring. They also offer varying degrees of application, system, and web monitoring too.

Icons of tools

Read More

Scalyr in the dark

Ying-yang symbol in black and whiteSometimes you want your dashboards to be dark.

And by dark, we mean dark backgrounds with light text and discernable colors. For many people who watch dashboards, looking for alerts and concerns, a darkened screen is more restful. And while research may have concluded that dark letters on a white background are easier to process, it also points out white backgrounds in a darkened environment may disrupt low light vision adjustments.

Scalyr dashboards are white, with dark text and colors in use to indicate the desired metrics. These dashboards are easy to read, understand, and clearly detail the tracked data.

Scalyr dashboard for Linux Processes with white background

However, it is reasonably easy to change this view by making use of the accessibility features in Chrome and Firefox. These reversed colors hack is not perfect as it impacts all of the sessions on the respective browser but quite suitable for a long-lived view or stable operations center.

Scalyr dashboard for Linux processes with black background

This hack is also not making use of the system-wide accessibility features, so only the browser is impacted. In short, the rest of your applications and background are viewable as usual, but the view of your browser has changed. Just be aware that other activity on your tabs may have rather wild results.

Chrome offers several accessibility extensions that allow some control over the look of a page presented in the browser. These extensions are available in the Chrome web store and are easily found via the Preferences (or Settings) page. The extension allows easy toggling of visual impact.

Firefox has built-in preferences that allow you to create the dark scheme. While it is easier than Chrome, requiring no extension installation, it does not provide an easy way to toggle off and on. It does have additional personalization features and does not change the data representation colors. (Note, an add-on exists that allowed toggling but is not supported in FireFox Quantum at this time.)

A similar capability exists on Safari, Windows Internet Explorer, and Edge but requires a CSS stylesheet insertion. I will cover those in a later blog.

The following directions are from my MacBook; however, the same workflow exists for all Firefox or Chrome browsers regardless of operating system. If you are having difficulty with your version, drop me a comment and I will try to help out.

Come to the Dashboard Dark Side. 

Step by step for Chrome:
  1. Open your Chrome browser.
  2. Click on the menu button.
  3. Choose the Settings or Preference menu item (alternatively, you could enter “chrome://settings” directly into the address bar).
  4. Scroll down the page and select Advanced.
  5. Scroll down to Accessibility and click on Add Accessibility Features. This click will open a new tab (or window).image cut of Chrome accessibility store
  6. In the Chrome store, find High Contrast.
  7. Click on Add to Chrome. This click will open a pop-up window. 
  8. Click Add Extension. High Contrast icon in Chrome browser bar
    A small icon will appear in Chrome in the upper right corner.
  9. Now the fun stuff. Click on the icon and Enable the extension. There are some cases where installing the extension automatically enables it. If so, just click the icon to bring up the selections.pop up window to control a11y settings
  10. Now that the extension is enabled click on Inverted Colors.
  11. To toggle back and forth, either make use of the keyboard accelerators or click the appropriate choice in the extension.
Step by step for Firefox:
  1. Open the Firefox browser
  2. Click the Menu button.
  3. Select Preferences or Settings (alternatively, you could enter “about:preferences” directly into the address bar).
  4. Scroll to Fonts & Colors (under Language and Appearance).Firefox Languages section of preferences
  5. Click on ColorsMenu of choices to change Firefox color selections
  6. In the Colors menu, change the associated colors.
    1. Text to White.
    2. Background to Black.
    3. Unvisited Links to Yellow (Optional).
    4. Visited Links to Light Blue (Optional).
      Please note that in Firefox, you can use what colors you would like. The above choices are offered as a starting point.
  7. In the Override the Colors box select Always.
  8. Click OK.

Reversing this choice requires entering preferences again and setting the Override the Colors choice to Never. As noted, the extension that offered a toggle button is not supported in Firefox Quantum.

So there you have it, a quick and dirty approach to getting your dashboards in the dark.

However, note that these changes will impact all tabs and windows of the browser, not just the dashboard views. For that reason, you may want to limit this use to a long-lived display or make use of the toggle capability of Chrome.

To learn more about accessibility in Chrome, check out Use Chrome with accessibility extensions – Google Chrome Help. For information about accessibility in Firefox, take a look at Accessibility features in Firefox. And to learn more about dashboards in Scalyr, check out how log analysis can lead you to needed actions.

If you try it out, tell me about your experiences in the comments.

 

Get Started Quickly With JavaScript Logging

Let’s continue our series on getting started with logging. We’ve already covered C#, Java, Python, Ruby, and Node.js. This time, we’re going to look at JavaScript logging. If you’re wondering how this differs from the Node.js article, this one will look at pure client-side JavaScript logging. Once again, we’ll get into

  • Logging in a very basic way
  • What to log
  • Why you should log at the client-side
  • How to log using a client-side JavaScript logging framework

JavaScript Shield with Scalyr Colors

Read More

How are you doing “observability”?

In today’s world of complex code and deployment, people on the DevOps front line face challenges in monitoring, alerting, tracing distributed systems and log aggregation/analytics. Jointly, these are often called “observability” (reference: Twitter blog).

hand drawn checklist with yes and no choices

These are challenges faced by multiple groups like DevOps, core engineering teams and Web 2.0 developers. We see concerns in web applications and traditional enterprise applications. We foresee even more issues in emerging spaces like IoT, event-driven design and microservices.

And as is usual in most complexity-bound problems, there are a lot of ways to solve these challenges. These include the use of discrete tools and procedural methods. Such approaches often cause gaps and leave edges uncovered. After all, the matrix of product types and usage (use cases) are large and growing and the need and scale are also increasing.

So, what do you think about observability? Scalyr is hosting a short survey to find what the current state of observability is. The survey looks are several areas, including tools and related issues and we invite you to chime in with your thoughts. While the survey is the best place to express your views, feel free to leave us a comment on this topic.

Please take the survey (at https://bit.ly/2JKezDb), and you can opt-in to get a copy of the results.