Sexy But Useless DevOps Trends

What’s sexy but useless? A Ferrari in a traffic jam. It’s beautiful, but all that power means nothing. When trapped in traffic, it can’t live up to its full potential.

Same with DevOps. While there are some critical DevOps functions that you absolutely need, there are some sexy but useless DevOps trends that are good to be aware of. Truth be told, there’s no recipe that will tell you how to succeed in DevOps. Everyone will have different opinions, and what worked for others might not work for you. But you can trust one thing: there are some actions that will guide you directly to frustration with DevOps.

With the amount of information out there about DevOps, you might get overwhelmed and think it’s not for you. You also might think the learning curve is too steep—that you need to change too many things before you get started. Maybe you’ll need a new team, new tools, more metrics, more time… you name it.

My advice is this: don’t get distracted by all things that people say about DevOps. These things I’m going to talk about here, for instance, are all style and no substance.

 

Like this Ferrari if it were stuck in a traffic jam, some DevOps trends are sexy but useless.

Read More

5 Critical DevOps Practices

DevOps is like pizza. We can’t think of pizza without considering critical ingredients: dough, sauce, cheese, and your preferred choice for vegetables and proteins. Everyone likes different toppings. In my case, I can’t think about pizza without extra cheese and meat. You might choose differently, but I think we can agree there are some ingredients that are critical for this food to be called pizza. Quality and ingredients will vary, but some things will always remain true.

Well, it’s the same with DevOps practices. There are some critical practices, and you can’t think about DevOps without considering them. Everyone will have preferred choices regarding the tools and the process, but the practice will remain and each practice complements the other.

Every critical DevOps practice takes time to get down, but the end result will be magnificent. So, let’s discuss what they are and how to implement them.

Pizza with Scalyr Colors

Read More

DevOps: Past, Present, and Future

While DevOps is no longer a brand-new field or movement, there continues to be rapid innovation in the space. Every day new tools are announced, new technologies are created, and teams around the world have to sort through the noise to figure out what’s actually important to their team and their environment. Some tools are transformative (VMWare), some are useful but quickly supplanted (LXC), some prove to be neat technology but never find wide adoption (VRML).

In this post I explore DevOps’ past, present, and future, reflecting on 2017 and looking to 2018 and beyond. What once was revolutionary is now commonplace, and the future holds the promise of much greater efficiencies, albeit with a significant amount of retooling to get there.

DevOps Adoption is: The Past

Today, DevOps is the de facto standard for modern IT and Operations. No longer are you an early adopter if you roll out DevOps. In 2017 only ~25% of companies hadn’t started down the DevOps path. In 2018 most companies will complete the journey, leaving those without DevOps in the minority. Is DevOps for everyone? I’d argue that the core concepts of DevOps— collaboration, communication, and efficiency—are beneficial to any company, in any industry.  The benefits that companies receive are both material and increase over time.

Through 2018 and beyond, DevOps will continue to entrench itself as the new normal in how companies build and run large-scale software systems. The business world will continue its adoption of the DevOps mindset and tools, occasionally crossing over from traditional software engineering into other practices, much as Toyota’s Lean manufacturing inspired the creation of Lean Startups and forms the foundation for next-generation hospital operations. The maturation of the DevOps space will drive the development of new terminology—for good reasons (more precise language and descriptions) and bad (our product is different because we target DevSecAIFooOps). These new labels (and inevitable jargon) will muddy the waters around DevOps a bit, mostly due to vendors fighting to stand out from the crowd and avoid becoming a casualty of industry consolidation.

The next phase of DevOps evolution involves marrying Agile product development processes with DevOps-centric production release environments. This workflow evolution promises to dramatically increase knowledge worker efficiency—but while very early implementers will start this process in 2018, the practice won’t become widespread until 2019 or later.

Containers are: The Present

Containers as a concept (be they Kubernetes, Docker, or some other new format) will start to “cross the chasm” and see much more widespread adoption in a variety of use cases. Most early adoptions will focus on enabling engineer productivity and software distribution. More evolved environments will use containers to incrementally deploy their microservices and ensure consistency among dev, test, and production environments.

Where the first wave of virtualization was all around efficient hardware utilization, the wave of containerized virtualization is about enabling consistent software environments. Expect to see more tools enabling software release orchestration using containers as the deployable code artifact. This will have benefits in terms of testability and reproducibility but will require teams to invest in new tools and modes of operation in order to make full use of the potential.

Increased adoption of containers will lead to a dilution of the core message and some industry pushback due to:

  • Fear of change
  • Valid attempts by engineers to separate fluff from reality
  • Lack of good tools and example use cases

Ultimately, containers will evolve into a core component of modern IT infrastructure much as virtualization did 15 years ago.

Unikernels are: The Future

Unikernels are a very promising bit of technology, but are still a couple years away from widespread adoption. For those not familiar with them, Unikernels are a newer type of container technology that embeds the operating system in your application (i.e., the reverse of your typical OS). Most operating systems need things like support for multiple users and applications, and, in most cases, a user interface. Unikernels ditch all of that overhead and pare the OS down to the absolute minimum. This is good from a security and reliability standpoint, but will require engineering and operations teams to retool to support monitoring, debugging, and deployment of Unikernel applications.

Unikernels have the potential to dramatically change the paradigms for production software environments (primarily due to dramatically altering the security landscape). In 2018 you’ll hear a lot more about them and vendors will start to tout their support—but expect to see only limited production adoption. My hope is that Unikernel standards will start to emerge, and at least one or two will be ready for early production deployments of common app platforms (UniK? IncludeOS? Xen?).

Hybrid Clouds are: Still to be defined

“Hybrid cloud” means many different things to many different people. To some it means running your environment across your physical datacenter and AWS/Azure; to others it means stitching together SaaS offerings from multiple providers; to still others it means only mixing of IaaS providers. The hybrid cloud story has been touted since the very early days of AWS with Microsoft, VMWare, IBM/Softlayer, Rackspace, and others all making their pitch to the market about what a hybrid cloud should look like. Often more than once.

The industry demand for hybrid cloud functionality continues to grow. Engineering and operations teams must be able to build, deploy, and run applications across multiple providers, and have those providers interoperate securely and efficiently. They need the ability to migrate selected IT workloads to cloud providers without undertaking massive retooling efforts. And yet there still doesn’t seem to be an agreed-upon set of solutions. Too many vendors are building walled gardens to keep customers in instead of building tools that allow seamless, secure, cross-platform communication. But, Microsoft, VMWare, Google, and others continue to invest in this area, so I’m hopeful we’ll start to see some type of consensus and standards developed over the next few years.  

I’ll be tracking the success of my predictions throughout 2018…and invite you to do the same. Commend me, call me out, or leave your own predictions in the comments below.

Log Management: What Is It and Why You Need It

To understand log management, you first need to understand what problem it solves. Once you see that, you’ll know both what it is and why you need it.

Software these days involves a lot of complexity that didn’t exist once upon a time. We’ve moved things into the cloud, created software/platforms/infrastructure as services, and embraced distributed computing.

That’s a sea change from the good ol’ days of the 1990s. Back then, you’d write a bunch of code, build it, put it on CDs or floppy disks, and mail it to people. It’s even a sea change from the 2000s, when the web application took over. Instead of CDs, you’d set up a web server, deploy your software to that, and let users and their browsers have at it.

But today, we have containers and microservices. We have software intelligence distributed around the globe, spinning up and down on demand, collaborating and orchestrating. We’ve traded the simplicity of the historical monolith for the flexibility and complexity of distributed intelligence.

Log Files in a Distributed World

Think about the change I’ve just described. And now imagine what that means for the existence of a log file.

In the 1990s, you’d add code to your application that dumped information to a single log file. If your users had problems, they could zip up that log file, along with an OS log file for good measure, and send those to you for troubleshooting. With 2000s web applications, that same application log file, along with the web server log file and the database log file, did the trick.

But now? Good luck. Your production operations include six RESTful microservices on six different servers, a bunch of on-demand containers, a few miscellaneous web apps, a service bus, and who knows what else? Each of those concerns is contained, isolated, simple, and useful.

But troubleshooting across those concerns, when the issue happens in the gaps, can be a mess. And gathering 20 different log files that you attempt to reassemble into some facsimile of order doesn’t help matters at all.

Log Management to the Rescue

That is where the idea of log management as a first class need enters the picture. If you have a desktop app or a simple web app, you can probably get by with grep, text editors, and elbow grease. But as soon as you grow beyond that, you’re going to need a better approach.

Log management is that better approach. Instead of regarding your applications’ logs as separate, unrelated entities, you conceive of them as parts of a whole. You weave them together and then use them to paint a dynamic, intelligent, and visual picture of the health of all your systems.

If that sounds daunting, don’t worry. You don’t need to implement all of this yourself. In fact, you definitely shouldn’t do it yourself any more than you should write your own source control. A lot of talented toolmakers have invested significant effort in helping you with your log management.

But rather than focus on specific tools, let’s take a look at log management as a function of its components. What does a good log management scheme involve, and what should you expect out of it?

Read More

Choosing Among Log Management Tools

When you google log management tools, an interesting thing happens. At the time of this writing, you see no fewer than 4 paid ads, followed by a series of posts. These include, and this is not a joke, a post that lists the top 47.  As a software developer and tools consumer, this drives me insane. It probably does the same for you.

An author named Barry Schwartz coined a term (along with an eponymous book) for this frustration. He called it “the paradox of choice,” and it describes how, while we like to have some choice and autonomy, too much paralyzes us. To understand this in simple, terms, imagine selecting music for a dinner party. If offered two albums from which to choose, you’d make a pretty quick choice. If offered hundreds, you might thumb through them for a long time, trying to consider the likely tastes of all of your guests. And you might actually just give up eventually, and opt for only conversation with no background music at all.

The Paradox of Choice Among Log Management Tools

Back in the DevOps world, you face a similar plight when trying to pick among log management tools. You understand that you need a better way to aggregate and mine your logs than “by hand, using Sublime Text,” so you start to do some research. And then, about two searches in, you find yourself staring at post entitled, “The Top 47 Log Management Tools.” And, if you’re anything like me, you rub your temples and say to yourself, “ugh, never mind, I’ll figure this out tomorrow.”

That, of course, lines up with Schwartz’s findings about human behavior. Beyond having a few options, each additional option presented to a group of people causes fewer people to participate. The higher the number of log management tools in those posts, the fewer people will actually pick any of them at all.

Luckily, there’s a path back to joy. And it’s not even terribly complicated. You just need to dramatically narrow the field.

So today, I’m not going to add to the pile of “pros/cons/features” posts out there comparing dozens of tools. Instead, I’ll speak to heuristics you can employ to help you choose among log management tools. I’m going to help you narrow the field from a paralyzing number of choices that you make you unhappy to a manageable number that empowers you.

Read More

Five Reasons You Need Log Monitoring

You probably regard application logging the way you think of buying auto insurance. You sigh, do it, and hope you never need it. And aren’t you kind of required to do it anyway, or something? Not exactly the scintillating stuff that makes you jump out of bed in the morning.

It feels this way because of how we’ve historically used log files. You dutifully instrument database calls and controller route handlers with information about what’s going on. Maybe you do this by hand, or maybe you use a mature existing tool.  Or maybe you even use something fancy, like aspect-oriented programming (AOP). Whatever your decision, you probably make it early and then further information becomes rote and obligatory.  You forget about it.

At least, you forget about it until, weeks, months, or years later, something happens. Something in production blows up. Hopefully, it’s something innocuous and easily fixed, like your log file getting too big. But more likely some critical and maddeningly intractable production issue has cropped up. And there you sit, scrolling through screens filled with “called WriteEntry() at 2017-04-31 13:54:12,” hoping to pluck the needle of your issue from that haystack.

This represents the iconic use of the log file, dating back decades. And yet it’s an utterly missed opportunity. Your log file can be so much more than just an afterthought and a hail mary for addressing production defects. You just need the right tooling.

Log Monitoring To the Rescue

I’ve talked in the past about one form of upgrade from this logging paradigm: log aggregation. A log aggregation tool brings your log files into one central place, parses them, and allows you to search them rapidly. But you can do even more than that, making use of log monitoring via dashboards.

Read More

The DevOps Job Market

DevOps as a profession and discipline just keeps growing in demand. This year alone, more than 100 global conferences are dedicated to DevOps — and even the “best of” lists are staggering. Hundreds of companies (including Scalyr) are developing tools specifically for the DevOps field, and blogs dedicated to everything from current news and trends to making light of DevOps daily frustrations abound (RIP DevOps Reactions). Even a casual survey of the trending-up of “DevOps” in Google search over the past five years makes it pretty clear that demand for professionals in this space will only continue to rise.

Point. Made. (via Google Trends)

Read More