Hello Everyone! We have what we have been told is not a complete ideal setup where we have searchable data for 90 days and retain archived data for a year. We commonly have troubleshooting efforts that occur or come up where the issue happened 6 months ago (outside our searchable data). This forces us to then do a log rehydration to get the data to be searchable out of our archives. There is a lot of down sides to this. 1. We only can submit those as admins (which is for the best I know) 2. They can take 6+ hours for the data to restore before anyone can proceed with their troubleshooting 3. We are limited to how much data we can restore at a time. Right now, we are hitting a situation where we have a large investigations going on which required rehydrating a lot of data and it now is preventing the rehydration of any other data for smaller investigations. We can't clear the larger data due to the investigation going on. We thought about looking at S3 Federated searches instead of going out to archives and dealing with log rehydration's. However, there is concerns this wouldn't really get us where we need to be either. We would love to look at other options. What are others doing to help in this process? How are you handling investigations outside your normal searchable date ranges? ... View more

The Problem We Were Solving Our organization runs Splunk Cloud with a mix of index families, a fairly complex multi-layer application architecture, and a user base that ranges from power users writing tstats queries to people who are brand new to SPL. The challenge wasn't just "can AI write a Splunk search" — it was: Can it write the right search for our environment? Can we trust that it won't kick off a runaway scan against a production index with 90 days of data and no scope filter? Can it know that realm= is the correct scoping identifier in our kube indexes but host= is what you want in our ivue indexes — and that these two values are not interchangeable? Out-of-the-box AI assistants don't know any of that. Without some assistance, our end users were struggling to get off the ground. We needed a way to encode it. What Claude Skills Are Claude (Anthropic's AI) has a feature called Skills — essentially structured knowledge files you attach to a Claude Project or Plugin. Skills are Markdown documents that the model reads as authoritative instructions and reference material. They're not training — they're live context loaded into every conversation. This matters because it means you can give Claude your organization's actual knowledge, rules, and constraints, and it will apply them consistently. You're not hoping the model "remembers" best practices from its training data. You're telling it, explicitly, what your environment looks like and how you want it to behave. What We Built: A Modular Skill File Architecture We built what a Claude Project with a set of skill files organized around a master router ( SKILL.md ) that tells Claude which specialized skills to load based on the type of request. The architecture looks like this: SKILL.md ← Master router ├── splunk-core.md ← Always loaded — safety rules, approval framework ├── splunk-indexes.md ← Index inventory, retention, "is this index active?" ├── splunk-search-writing.md ← SPL patterns, macros, field reference ├── splunk-dashboards-viz.md ← Dashboard Studio JSON generation ├── splunk-member-investigations.md ← Member scoping, lookup workflows ├── splunk-operations.md ← Alerts, reports, rehydration processes ├── splunk-training.md ← Adaptive onboarding for new users └── references/ ├── connector-limitations.md ← MCP connector quirks and guardrails ├── cross-layer-investigation.md ← index pivot workflow └── user-token-setup.md ← Token provisioning guide Each file is scoped to a specific domain. The router tells Claude which combination to load based on what the user is asking. Asking to write a search loads splunk-core + splunk-search-writing . Investigating a member issue loads splunk-core + splunk-member-investigations + splunk-search-writing . Building a dashboard adds splunk-dashboards-viz . We also are rolling this into a plugin to use for Claude Code as well. From this we are aiming for two parts: Part 1: Guardrails on Best Practices Part 2: Encoding Our Splunk Structure (what is the purpose of an index or source type) We are running the Splunk MCP server here connected to our Claude system. I won't lie it took a bit to manage the initial setup to setup each user with individual tokens in the MCP server compared to other companies in which allow end users to manager their own tokens without gaining access to global admin settings. All in all, this has allowed our users to go from basic users to more informed and efficient users. Their confidence is growing as they learn from the training aspect of this setup. We are still in the testing phase, but for us it is a game changer. Curious how many other Splunk users are exploring this part of the AI world! ... View more