Shock the Chaos Monkey

July 17, 2023 by Jay Cuthrell

Share and discuss on LinkedIn or HN

This week’s musical inspiration in title and lyrics:

This week we take a look at chaos engineering adoption within platform engineering teams.

Getting Informed

One could say that Murphy’s Law about _things that can go wrong_ is a precursor to chaos engineering. However, thinking and acting upon what might (will?) go wrong is just part of the discipline known as chaos engineering.

Now it’s time for reading 📖, watching 📺, and listening 🎧 suggestions:

📖 - Blog posts

First, 📖 AWS’s recent take on chaos engineering from Seth Eliot, Jason Barto, and Laurent Domb has me thinking about the “easy button” to facilitate effortless product engineering.
Second, 📖 The New Tech Stack for Building LLM Powered Applications from Sandi Bensen has me thinking more about the chaos potential in AI scaffolding.
Third, 📖 The Chaos Data Engineering Manifesto: Spare The Rod, Spoil Prod from Shane Murray brings the tasty memes to _inject data_ into chaos engineering (pun intended).
Fourth, 📖 Designing for a better PLG interaction from Greg Meyer reminds us to “anticipate and welcome chaos”

🎧 - Podcast episodes

First, 🎧 A Framework for Making Sense of AI Chaos from Brian Gracely brings more perspectives on AI at this point in time.
Second, 🎧 Chaos Engineering with Uma Mukkara from Jordi Mon Companys and Uma Mukkara is a great deep dive thinking about and seeking greater resiliency.

📺 - Videos

First, 📺 S4E21: Kelly Shortridge - Security Chaos Engineering & Resilience (mash that subscribe button!) in which Dr. Nikki Robinson and Chris Hughes interview Kelly Shortridge for a wide ranging discussion from empathy to economics to ego to leveling up — and a surprise chaos guest! 🔇😼😹
Second, 📺 “Tom Brokaw Pre-Tapes” that aired on Saturday Night Live in 1996 aka my favorite chaos engineering analogy.

You can ape the ape 🎶

In 2010, Netflix Engineering blogged about Chaos Monkey. Thirteen years later, chaos engineering has become part of the lexicon for platform engineering teams.

When an audience asks me about chaos engineering, I am reminded of a comedic analogy^[1]. The comedic analogy helps provide an understanding of the need for planning that anticipates highly probable events that impact the performance or availability of elements in a system.

I also tend to ask the audience one question to frame the discussion:

How confident are you that your top SRE or top response team member or entire platform team could take an extended vacation where everything that could happen is something the rest of the remaining software development team can handle on their own?

Oh, the answers… so many answers.

Bottom line: Convergence of SDLC disciplines is ripe to enable progression to product engineering mindsets and product-led growth companies.

And the news is breaking 🎶

If you are having Fudge Sunday déjà vu — it is warranted. Chaos engineering was covered back in 2021 in Fudge Sunday #113 for several hyperscale public cloud service providers as the last issue of a five (5) part series on “Cloud in Public”.

https://fudge.org/archive/fudge-sunday-cloud-in-public-mean-time-to-rca

https://fudge.org/archive/fudge-sunday-cloud-in-public-impact-mapping

So, what will be the next big thing in chaos engineering within platform teams?

Until then… Place your bets!