Why Are Engineers Getting DevOps Fatigue?

As an engineer, you already have enough responsibilities when developing software. Adding more tasks–say, DevOps-related ones—to your workday activities might not sound very appealing. With DevOps, not only are you responsible for producing working software, but now you also need to automate the building, testing, and deployment phases of the software. That’s a lot to take care of! But the extra work aside, maybe you’re just tired of the DevOps movement, and all the hype surrounding it is causing DevOps fatigue.

As a former developer, I can identify with that feeling of fatigue. I’ve also seen some colleagues reach a certain level of frustration with DevOps. There are times when we make the mistake of taking on everything, even the releases. This is especially common if we’re perfectionists and don’t like to deliver software with bugs. We could even get to the point of releasing our code to production. (Although now that you’re “doing” DevOps, that might become your responsibility anyway.) After all, if we code it, we know the things that could go wrong and how to fix it if there are problems.

Even though now I’m more on the operations side of things—dealing with servers and helping companies implement DevOps—let me share some thoughts with you about why I think engineers are getting DevOps fatigue.

devops_fatigue_scalyr
Read More

Kubernetes: The Next VMware?

It’s been almost 10 years since VMware started selling ESX version 4.0. This set the path for VMware to dominate more than 75% of the virtualization market in 2017. Gartner considers this market “matured” since most of its revenue comes from maintenance instead of new licensing. Many companies have consolidated their workloads with virtualization, but there are new problems to solve.

Delivering, testing, deploying, and scaling applications are among these challenges. Teams that implement microservices also need to automate as much as possible to make them manageable. Kubernetes, Marathon, Swarm, and Nomad compose a new breed of tools that respond to these needs through orchestration. If you host on-premises or in the cloud, consider them to help your business more quickly deliver code to production.

Companies evolving towards data-driven decision-making often implement machine learning and business intelligence tools, looking for an edge in their markets. As information technology professionals, it’s our responsibility to make sure our businesses select tools that

  • perform in a reliable way;
  • allow quick deployment of new features;
  • scale properly in response to user demand; and
  • deploy new software in a safe and reproducible way.

In this article, I explain why I think Kubernetes is a market leader in the orchestration space and how it might steal VMware’s thunder in the not-so-distant future.

kubernetes_the_next-vmware_scalyr

Read More

Modern software is great, but when it comes to observability, gird your loins!

This won’t come as a shock, but engineers who are on the hook to develop revenue-generating software are quickly moving away from traditional, monolithic architectures and delivering code faster than ever. This trend is especially evident among Scalyr customers, but we wanted to understand how it is generally unfolding and how it affects DevOps observability. We surveyed 155 software development practitioners in DevOps-focused organizations over the last couple of months, and just released our State of DevOps Observability Report. You can download the full report here or check out its summary in this infographic, but in this blog I’ll give you the Cliff’s Notes version of what we found.

Croped infographic on DevOps Observability

We confirmed that organizations really are shifting away from traditional, monolithic architectures. Three-quarters of survey respondents say they deliver at least some of their applications, and more than one-third deliver most of their applications, as microservices. They are also delivering software rapidly, with 71 percent of engineers pushing code into production at least weekly and nearly one-third doing so at least daily. Looking at all of our survey findings through the lens of these two trends, we realize that modern software delivery is putting pressure on DevOps observability.

Here are some of our findings:

Companies are delivering software in a modern way.

  • Three-quarters of respondents deliver some and more than one-third deliver most of their applications as microservices.
  • 71 percent of engineers push code at least once per week, and nearly one-third push code at least once per day.

Read More

How Serverless will Change DevOps

This post on Serverless is from guest author Limor Wainstein, email is limor@agileseo.co.il.

During the past decade, both DevOps and serverless have become quite popular for software development. These not only challenged the way software is being developed but also affected the entire software development lifecycle. First of all, let’s try to understand serverless and DevOps are.

Serverless and DevOps

Serverless is commonly used in conjunction with terms such as architecture, computer, and services. In brief, serverless architecture refers to the solution architecture of software applications built using fully managed third-party services (serverless services) as core dependencies. At its core, serverless computing provides runtimes to execute code, which is also known as function as a service (FaaS) platforms.

DevOps is a collection of practices and tools that increase the ability to deliver (build, test and deploy) applications and services more efficiently. DevOps requires more automation and eliminating overhead in setting up the infrastructure as well as in the build and deployment of code. DevOps does not only affect the technology; DevOps is also targeted at people and team structures.

Continuous Integration (CI) and Continuous Delivery (CD) tools are used to build, test and deploy code and are often done using build automation  Tools that provide fast resolution of errors and bugs are equally crucial.

In addition to using FaaS, fully-managed Docker containers as a service are also evolving. For example, technologies such as AWS Fargate allow to run these Docker containers and scale them without needing to provision and manage any underlying infrastructure. This is useful for DevOps since the containers can also be used to build, test and deploy applications.

Fully Managed DevOps Services

Fully managed DevOps services are beginning to emerge for CI and CD where these services are offered in software as a service (SaaS) models, or hosted services like Travis, Circle CI where the respective third-party service providers manage the underlying infrastructure and services. These services could range from semi-managed to fully managed serverless services that are used for the core of DevOps.

Docker containers underly many of these managed services, and these containers are used to allow isolated and customized build environments. Some services such as AWS Code Build enable defining the Docker image together with the required libraries and tools. This allows direct execution of the build job without needing to reinstall for each build. Various orchestration tools are used by these managed services, such as Kubernetes, Docker Swarm, AWS ECS, to control the fleets of containers.

DevOps Tools with Serverless.

There is also a tendency towards implementing DevOps using serverless architectures. The ability to implement unique and efficient CI/CD tailored to the application requirements, consumption-based cost models, supportive services (e.g., AWS Code Build, AWS Code Pipeline, etc.) create attraction towards this approach. By using serverless services, it is possible to implement an entire build, test and deploy pipeline by writing the glue code using serverless services, without using any hosted solutions.

Open Source and Serverless

Over the past few years, many open source DevOps tools started emerging. One of the most popular open source DevOps tools is the Serverless Framework.

The Serverless Framework allows simplification by:

  • Declaring the microservices infrastructure, endpoints and mapping it with execution logic (for example, with a Lambda function).
  • Building artifacts, archiving and deploying to infrastructure.
  • Extending DevOps capabilities using plugins.

Cloud providers are also beginning to develop similar tools. For example, AWS recently introduced its AWS Serverless Application Model (SAM) which could potentially replace the Serverless Framework functionality when it comes to declaring the serverless APIs. Contributions by the open source community do keep the Serverless Framework supporting newly-introduced AWS services as they appear.

In addition, open source security is becoming a challenge when using third-party libraries. On the other hand, vulnerability scanning is also becoming a commodity, especially with the introduction of the NPM scanning process. This has helped to reduce the development risks in building applications.

Infrastructure as Code

With the adoption of serverless architectures, infrastructure provisioning has become a significant part of the deployment process. Applications are using different serverless services as building blocks, so it is crucial to be sure that these services are correctly configured and connected. One of the best practices is to define the infrastructure in the code using declarative tools and languages like Terraform. This enables versioning of the infrastructure along with the code thus reducing the manual work on the DevOps process.

Immutable Infrastructure

Another effect of serverless for software application development is how we look at the serverless services modifications. It is often possible to provision new instances of serverless services while keeping the existing instances running. This means you can deploy new instances of serverless services and switch to the more recent version upon success. Serverless has made Blue-Green and Canary deployments both more practical and affordable for various applications.

Distributed DevOps

Using serverless technologies for application development means that DevOps has to deal with distributed systems. While it’s possible to use different serverless services from different providers, configuring a CI/CD pipeline involves coordinating these operations. This adds further complexity to the DevOps processes. It is important to implement more visibility into each serverless service with detailed monitoring when these services are provisioned once a change had been made.

Summary

Overall, we’ve outlined a few ways in which serverless is changing DevOps practices used for software development. To wrap up, let’s look at several best practices for serverless and DevOps.

  • Implement automation for CI/CD.
  • When implementing tests such as code style, unit tests, integration tests, UI tests, make them a part of development routine checks.
  • Upon committing the code, try to run the code style checking using git lifecycle hooks if possible.
  • Automate as needed to support building the artifacts, running all the test cases and send the results back to Pull Request using a CI tool.
  • Rerun the tests before deployments, preferably through an appropriate deployment trigger.
  • Include the infrastructure in the code.
  • Use managed services for CI/CD whenever possible to reduce the overhead of infrastructure management.
  • Design an application for rollback support for both previous versions and database migrations.
  • If it’s a database change, create scripts that can both apply and revert the changes.
  • Create version and store deployments in a durable place.
  • Allow deploying specific versions of application deployment.
  • Choose tools that help streamline the debug and monitoring cycle.

Separating the deployment cycles of serverless projects allows teams to make changes without redeploying the entire application. It remains vital to learn and understand the changes introduced by DevOps and serverless process and technologies and to adapt accordingly so that the DevOps processes are aligned to support serverless applications.

Bio:
Limor is a technical writer and editor at Agile SEO, a boutique digital marketing agency focused on technology and SaaS markets. She has over 10 years’ experience writing technical articles and documentation for various audiences, including technical on-site content, software documentation, and dev guides. She specializes in big data analytics, computer/network security, middleware, software development and APIs.
Twitter: @LimiMaayan

The 10 Commandments of Logging

When you search for things on the internet, sometimes you find treasures like this post on logging, e.g. creating meaningful logs.

This post, from a few years back, is authored by Brice Figureau (found on Twitter as @_masterzen_). His blog clearly shows he understand the multiple aspects of DevOps and is worth a visit.

In recent discussions with Brice, he stated that he believes the rules are still quite valid with some tweaks needed for containers and Kubernetes and that in today’s world of GDPR, logging sensitive data is a non -starter. But these rules are informative and worth consideration.

Our thanks to Brice for letting us repost his blog under Creative Commons CC-BY.

Guest author Brice Figureau

2 stone tablets with Roman numerals from 1 to 10
Copyright: moises / 123RF Stock Photo

After writing an answer to a thread regarding monitoring and log monitoring on the Paris DevOps mailing list, I thought back about a blog post project I had in mind for a long time.

I wrote this blog post while wearing my Ops hat and this is mostly addressed to developers.

Knowing how and what to log is, to me, one of the hardest tasks a software engineer will have to solve. Mostly because this task is akin to divination. It’s very hard to know what information you’ll need during troubleshooting… That’s the reason I hope those 10 commandments will help you enhance your application logging for the great benefits of the ops engineers 🙂

1. Thou shalt not write log by yourself

Read More

How are you doing “observability”?

In today’s world of complex code and deployment, people on the DevOps front line face challenges in monitoring, alerting, tracing distributed systems and log aggregation/analytics. Jointly, these are often called “observability” (reference: Twitter blog).

hand drawn checklist with yes and no choices

These are challenges faced by multiple groups like DevOps, core engineering teams and Web 2.0 developers. We see concerns in web applications and traditional enterprise applications. We foresee even more issues in emerging spaces like IoT, event-driven design and microservices.

And as is usual in most complexity-bound problems, there are a lot of ways to solve these challenges. These include the use of discrete tools and procedural methods. Such approaches often cause gaps and leave edges uncovered. After all, the matrix of product types and usage (use cases) are large and growing and the need and scale are also increasing.

So, what do you think about observability? Scalyr is hosting a short survey to find what the current state of observability is. The survey looks are several areas, including tools and related issues and we invite you to chime in with your thoughts. While the survey is the best place to express your views, feel free to leave us a comment on this topic.

Please take the survey (at https://bit.ly/2JKezDb), and you can opt-in to get a copy of the results.

DevOps Security Means Moving Fast, Securely

In this world of lightning-fast development cycles, MVPs, and DevOps, it may intuitively feel like security gets left behind. You might be thinking, “Aren’t the security guys the ones who want to stop everything and look at our code to tell us how broken it is right before we try to deliver it?” Many feel that DevOps security is a pipe dream.

Is it possible to be fast and secure? Lately, I’ve been drooling over a sports car—namely, the Alfa Romeo Giulia Quadrifoglio. Long name, fast car. It holds some impressive racing records and sports 505 horsepower but also is a Motor Trend Car of the Year and an IIHS Top Safety Pick. These awards are due to automatic braking technology, forward-collision warning, lane-keeping assistance, blind-spot monitoring, and rear cross-traffic alert. It is possible to be fast and safe.

The key to DevOps security is to move forward with development. Security teams need to understand why DevOps practices are so effective and learn to adopt them.

Man Running Fast with Scalyr Colors

Read More

DevOps Engineer: What Does It Take to Land the Job?

Every day, more and more companies are looking for people with DevOps knowledge. As I’m writing this post, there are 49,000 results on LinkedIn for DevOps jobs. That’s a lot of job openings.

The high demand for DevOps engineers must be for a reason, right? Well, according to devops.com’s “The State of DevOps Adoption and Trends in 2017” report, DevOps adoption has increased in the last couple of years, especially since 2016, and it won’t stop. The problem is that some organizations are finding it hard to start the journey.

Although starting a DevOps initiative is not an easy task, looking for people that have the necessary skills is even harder. It doesn’t matter what your background is; developer or sysadmin, you’ll need to improve or acquire some new skills and knowledge to succeed in your day-to-day job.

Let’s explore what you need to land a DevOps job.

Microscope over resume with Scalyr colors

Read More

Growing a High-Performance DevOps Culture

Culture is one of those things where we all know what it is but can’t explain it. Well, according to Wikipedia, culture is “the social behavior and norms found in human societies.” But in simple words, it’s all about people: how they interact, how they behave, how they talk, and what they practice. And culture is the foundation of a successful implementation of DevOps.

John Willis, an established speaker and writer on the subject of DevOps, coined the term CAMS (culture, automation, measurement, sharing) at a talk where he explained that DevOps culture is about breaking down silos. But what I find most striking about his discussion of culture, as summarized in the DevOps Dictionary, is the observation that “fostering a safe environment for innovation and productivity is a key challenge for leadership and directly opposes our tribal managerial instincts.” So the starting point for your DevOps journey is good leadership. After that, it’s just about how to grow your team to become a high-performing one.

A high-performing team in DevOps, according to recent research, is one that

  • Does deployments often, meaning several times a day.
  • Delivers a change with a fast lead time (minutes) after it’s been pushed to a shared repository.
  • Has a short (again, minutes) mean time to recover (MTTR).
  • Has a small change failure rate (described here).

So how do you grow a high-performance DevOps culture? You create a culture that will produce a team that delivers on time with confidence in a predictable manner. Here are the things that will help you get there.

High performance gauge with Scalyr colors

Read More