You step up to the machine. The pixelated corridors of a certain 1993 FPS load in front of you, EMP Pulse Glove already in hand. But this isn't nostalgia, every snarling robot prowling the map is a live Kubernetes pod, and behind it sits a fully functional e-commerce web store humming along in real time.  You pull the trigger.  A pod dies. Somewhere in a running cluster, a piece of a working application just vanished. Take aim again and you're not just fragging robots you're spiking CPU, choking the network with latency, and generally playing the role of the worst thing that can happen to a production system: pure, gleeful chaos. It's Doom, reimagined as a chaos agent. And the best part? You get to watch a real observability platform catch you in the act.  The Chaos Under the Hood  Here's what's actually happening while you're having fun.  Every "monster" on screen is mapped to a real pod running an OpenTelemetry demo e-commerce app inside a live MicroK8s cluster. When you take a shot, the game's Unity command bridge fires a real Kubernetes API call to terminate that pod. Kubernetes does what Kubernetes does, it notices the missing pod and spins up a replacement. But for a moment, part of that web store is genuinely gone.  Want more mayhem? The game hooks straight into the open source project Chaos Mesh to inject engineered failures:  Network latency — NetworkChaos experiments slow the connections between services, mimicking a degraded network.  CPU and memory stress — StressChaos piles pressure onto target pods, simulating a resource crunch under load.  Pod failures — pods get killed outright, testing how the system recovers.  None of this is smoke and mirrors. You are performing legitimate chaos engineering. The same discipline SREs use to deliberately stress systems. Except the trigger is a EMP weapon instead of a YAML file. And the whole time, a hybrid observability pipeline is watching: traces and metrics flow to Splunk Observability Cloud via the OpenTelemetry Collector, while event logs stream into Splunk Enterprise through the HTTP Event Collector.     Breaking things is fun. Understanding what breaks and why is the point.  When you kill that pod, the interesting question isn't "did it die?" — it's "how fast did we see it, and how quickly did the system heal?" That's the engineering takeaway KubeDoom drives home for admins and DevOps teams:  Resilience is observable. You can watch Kubernetes self-heal in real time pods rescheduling, services recovering  and measure exactly how the application responds under duress.  Chaos without visibility is just an outage. The difference between a controlled experiment and a 2 a.m. page is instrumentation. With traces, metrics, and logs correlated in one place, the "why did everything go sideways?" question actually has an answer.  Splunk turns panic into a dashboard. As you spike CPU or inject latency, you can flip to Splunk and see the impact land the spans, the metric spikes, the pod events proving that when a real incident hits, you'll have the same clarity instead of guesswork.  That's the whole idea: give people permission to break things, then show them the tooling that makes breaking things safe to learn from. It's the fastest, most memorable way to understand system resilience and it's exactly why the KubeDoom lounge is worth the trip to the show floor.  P.S. There are five full levels of Chaos, and Yes, there is a final boss if you can make it that far... Come find the KubeDoom lounge at .conf26. Bring your worst. Splunk will catch all of it. Register Now!   Explore all of our .conf26 spotlight articles here   ... View more

I’ve been having a lot of fun learning about Kubernetes and Observability. I set myself an interesting challenge: could I streamline the deployment process for the open-source OTel-Demo application alongside the Splunk OpenTelemetry collector to make it both simple and efficient?  Keep in mind, I’m not a developer. Since the world of Kubernetes (K8s) is largely developer focused I knew this would be a challenge for me to learn some new tricks. I also wanted leverage the cheapest & smallest hardware I could run this demo application on, so anyone could simply follow this Lantern Article and quickly deploy a Kubernetes application that’s been instrumented for Open Telemetry, and the Splunk OTel Collector all without knowing how to write any code. For hardware I selected the Raspberry Pi 5, on this PI we'll leverage MicroK8s, a lightweight Kubernetes distribution, and Helm to deploy a demo application that streams critical metrics, traces, and logs to Splunk Observability Cloud. To deploy this demo, you simply need to follow the step by step instructions in this Lantern Article.  This Lantern Article  will walk you through setting up your Raspberry Pi 5, installing MicroK8s, and deploying a demo application using some YAML values files I have stored on my GitHub repository to quickly get started with Splunk Observability Cloud. These YAML files allow the OTel-Demo application to be setup with the Splunk OTel-Collector making it easier for you to deploy and get instant results. No changes to the source code or learning how to change any settings in the demo applications. If you already have a Linux host that is setup for Kubernetes, simply skip down to Step 3 in the Lantern Article, and the deploy the Helm charts, just pay close attention to the YAML values files I have created for those Helm charts, as they do all the heavy lifting to make sure its deployed with the proper configuration to demo Splunk Observability Cloud. If you successfully deploy it, let me know by commenting below!   ... View more

Introduction to Splunk Observability Cloud - Building a Resilient Hybrid Cloud  In today’s fast-paced digital landscape, applications are supported by a growing array of technologies, all designed to accelerate time-to-market. However, the competition is fierce, and failing to move quickly could mean being outpaced by rivals. Yet, operating at this speed is anything but straightforward. The environments businesses utilize grow increasingly complex, often spanning private data centers, public clouds, and colocation facilities. Among ongoing upgrades, migrations, and innovations, one truth remains: customers typically don’t care about where a business is in its transformation journey; they expect seamless service. Each component of your infrastructure generates immense volumes of data. Logs from applications, servers, firewalls, network performance tools, and security monitoring have created an endless amount of information. When issues arise, determining where to look for the cause can be overwhelming. At Splunk, we understand this challenge and aim to cut through the noise, providing clarity and actionable insights. Without the ability to correlate an application’s performance with business outcomes and user experience, organizations are left in the dark. The Evolution of Application Architecture I’ve been building enterprise IT infrastructures for over 20 years now. Looking back in the early 2000s, classic 3-tier application architecture was much simpler. A new application typically required merely deploying a physical or virtual server, installing an OS, and layering the application on top.   Troubleshooting involved checking the OS performance and logs. However, these straightforward architectures couldn’t scale or change fast enough to adapt to the evolving digital landscape. The advent of distributed application architectures, such as Kubernetes and microservices, revolutionized how applications are built and deployed. Modern applications are now interwoven across multiple clouds and can be developed quickly. However, as these applications grew more complex, they often emerged from shadow IT teams that rapidly turned them into mission-critical operations requiring dedicated monitoring and support. Classic infrastructure teams that were built around monitoring 3-tier applications were now tasked with monitoring these new distributed architectures. Traditional tools and processes designed for 3-tier applications do not suffice for these new distributed architectures. When errors occur, teams often find themselves struggling across various siloed tools, resulting in costly “tiger” teams and diminished end-user satisfaction. This is where Splunk Observability Cloud comes into play. Introducing Splunk Observability Cloud Splunk Observability Cloud empowers organizations to manage and monitor both classic 3-tier architectures as well as modern distributed applications with speed and precision. By leveraging the power of our AI engine, Splunk helps businesses identify problems faster and significantly reduces mean time to resolution (MTTR). Thus building a more resilient architecture. Let’s take a closer look at how Splunk enhances observability through foundational features tailored for both 3-tier and modern architectures. Today, modern application services may be distributed across private data centers, hyperscaler clouds, or various third-party SaaS offerings. Each service operates with its own tools and access procedures, making it challenging to pinpoint errors. Splunk, however, provides visibility across all these services. With our Observability Cloud, we can aggregate logs, performance metrics, and application traces, offering a comprehensive overview of how each service interacts. Splunk application trace analysis reminds me of classic Windows or Linux network tracing tools like trace route, Trace route would show an admin each network hop from point A to point B. Splunk Application Traces powered by OpenTelemetry work in a similar fashion, this would allow teams to visualize the communication between application services. This interplay creates Splunk Observability Cloud application service maps. These service maps are dynamically created and illuminate error locations and their impacts on upstream and downstream services. Moreover, our AI capabilities extend to predictive analytics, enabling early anomaly detection. By providing real-time alerts on performance issues, teams can address potential problems before they escalate. The Splunk Observability Cloud UI streamlines this process, allowing for rapid examination of logs tied to specific services through the Splunk Platform.  Bringing all these pieces together gives you a powerful view of the big infrastructure picture of your entire hybrid cloud. Helping you reduce the amount of tools necessary to diagnose issues in your environment. While also, reducing your MTTR across all your public and private cloud infrastructure. Conclusion In this blog, I’ve only touched lightly on what Splunk Observability Cloud is. Many fans of Splunk know us for our log analytics, but Splunk does so much more than just logs. We can use the power of our log analytics (Splunk Platform) along with Splunk Observability Cloud to monitor the performance and diagnose issues of an entire hybrid cloud all with AI.  This is my first blog as a new Splunker, I ask you to continue on this journey with me. In each blog, I’ll dive deeper into what Observability Cloud is and all the features and capabilities it has to offer. You’ll soon discover Splunk is more than logs. Want to try it out? Take Splunk Observability Cloud for a free 14 day trial.  https://www.splunk.com/en_us/download/o11y-cloud-free-trial.html ... View more