When your problem is impossible, redefine the problem.
In an earlier article, I described how Scalyr searches logs at tens of gigabytes per second using brute force. This works great for its intended purpose: enabling exploratory analysis of all your logs in realtime. However, we realized early on that some features of Scalyr―such as custom dashboards built on data parsed from server logs―would require searching terabytes per second. Gulp!
In this article, I’ll describe how we solved the problem, using two helpful principles for systems design:
- Common user actions must lead to simple server actions. Infrequent user actions can lead to complex server actions.
- Find a data structure that makes your key operation simple. Then design your system around that data structure.
Often, a seemingly impossible challenge becomes tractable if you can reframe it. These principles can help you find an appropriate reframing for systems engineering problems. (more…)