We’ve all been there, right? Living our best life, working on our Splunk environment, when out of nowhere a user files an incident about an index having no data for the last 2 weeks! Scrambling to get it remediated, potentially without the ability to backfill the missing 2 weeks, is not what anyone needs in their life.
I’m here to help you become more proactive in monitoring so that data outages can be noticed and remediated faster. As a manager of the Splunk Security Center of Excellence Search team, it is our responsibility to ensure data is flowing as expected, and these are the processes our team follows today. While the solution I will lay out is not the right solution for every team, it will hopefully provide the right building blocks or ideas so that new and better solutions may follow!
Short on time? Here's the whole post in four bullets:
The Setup: A handful of CSV lookups (one each for index, host, sourcetype, and source) define how often each piece of data should show up, and who to bug when it doesn't.
The Deployment: Four saved searches run every 10 minutes, compare actual ingestion against those expected frequencies, and page the right people via messaging tool, paging tool, or email when something's overdue.
The Upkeep: The lookups live in version control, and a second layer of self-checking alerts makes sure nothing falls through the cracks, which includes catching new data sources that were never added to the lookups.
The Results: Across roughly 90 actively monitored indexes (plus dozens more tracked at the host, sourcetype, and source level), we caught 36 outages in a recent 30 day window alone. We were able to fix outages well before any of them turned into a 2-week surprise that a user noticed.
Note: This article will focus on full outages for an index, sourcetype, host, or sourece, not partial outages or latency issues.
The Setup
The Lookup Basics
It all starts with something delightfully low-tech: a CSV lookup file. (A KVStore lookup works too, if you're feeling fancy.) For our environment, we monitor data in four levels: index, host, sourcetype, and source; and therefore, we have four distinct lookup files.
Our index lookup, let’s call it index_monitoring.csv, is comprised of the following base fields:
Index
Expected frequency (in seconds)
Data owner (team)
Subject Matter Experts (SMEs)
Criticality
Paging key in case of escalations
Runbook URL for the dataset, if one exists
Time ranges to filter out – this allows us the ability to not trigger an alert for an index if we know that it will likely have an outage due to a recurring maintenance window, for instance.
However, we extend this to include many more fields, such as which ticket number(s) were for onboarding the data, if we’d like to exclude the monitoring for any specific reason, and email addresses if anyone prefers email alerts.
We keep similar fields for our other lookup files, however we do not fill them out if the information is the same at the index level (if the same team owns the index as the sourcetype, we do not need to keep it updated in both locations, just the main index file). We also do not monitor every index at the more granular levels, only a subset determined by both the data owners and the admins. Mostly, if sourcetypes, for instance, have different ownership or ingestion methods, we might want to monitor them separately than just at the index level. The other thing to consider is that, for us, we have determined that if we monitor an index by sourcetype, host, or source, we will not monitor it at the index level. This reduces noise: if an index goes down, every sourcetype alerts, so an index-level alert would not be necessary. That said, we require that once we start monitoring by a sourcetype, host, or source, we will add every sourcetype, host, or source within that index to the lookup, whether it needs to be excluded from monitoring or not. This way we have a full inventory of the metadata inside an index in case we do ever need to turn on monitoring.
Here is an example layout of our lookup files (not all of the fields listed above appear in the examples below, to save space):
index_monitoring.csv
index
frequency_s
owner
sme
critical
exclude_time
paging
exclude
index_a
3600
IT Team A
names
P2
1
index_b
600
Security Team B
names
P1
insert_key
index_c
86400
IT Team C
names
P3
(day="Fri" hour>=23) OR day="Sat" OR day="Sun" OR (day="Mon" hour<=18) OR Holiday=*
index_d
7200
Security Team B
names
P4
sourcetype_monitoring.csv
index
sourcetype
frequency_s
owner
sme
paging
exclude
index_a
st_a
600
insert_key
index_a
st_b
300
Security Team B
names
insert_key
1
index_a
st_c
3600
Security Team C
names
insert_key_new
Any blank field in sourcetype_monitoring.csv will use the field value from index_monitoring.csv for the matching index, except for exclude. That's why index_a can be excluded (exclude = 1) while its sourcetypes are still monitored. We keep this same format for source and host lookups, as well.
Pro tip: Use wildcards in lookups if any metadata has repetitive values. This happens a lot when a source, for instance, has a datestamp in the directory path. Wildcard lookups will require a definition in transforms.conf like the following:
[source_monitoring] batch_index_query = 0 case_sensitive_match = 1 filename = source_monitoring.csv match_type = WILDCARD(source)
The Exclusions
Take another look at the exclude_time value on index_c. It's a search condition checked against the current day and hour each time the alert runs, and it says: don't alert from Friday night through Monday evening, or on a holiday. The Holiday=* piece works with a separate holiday lookup comprised of the date, holiday name, and an epoch date for start and end.
Pro tip: Watch your timezones. A holiday lookup with just a date can silently misfire if the holiday date is not in the same timezone as the server running the search, which is why we include those epoch times.
The Thresholds
How did we determine the initial values for the frequency expected for each index, sourcetype, etc.? We ran a tstats command for each index, using _indextime, to determine how frequently the data was ingested and took the 90th percentile of index-time delta to start with. Upon initial creation, or addition of new indexes, this may need to be adjusted more often to reduce false positives, but p90 should present a reasonable estimation for expected windows.
Pro tip: Build in a little buffer. If a script usually runs daily at noon but occasionally comes in a few minutes late, set your expected frequency to just over 24 hours instead of exactly 24, otherwise you'll page yourself over nothing.
Due to the nature of the initial setup, it is expected to have noisy alerts while tuning thresholds. We suggest either not deploying an alert action until tuning has been completed, or sending the alerts to a testing environment (a test email, test direct messaging, test on-call tenant, etc.) instead of deploying to production. If writing all results to a summary index, you’re able to monitor all of the noise until it reaches a more sustainable and accurate pace.
Pro tip: After having an established and tuned threshold for monitored datasets, use caution when adding alerting for new datasets. We let the data ingest without alerting for a short period of time so we can establish a baseline threshold before turning monitoring on for it.
The Deployment
Armed with the lookups, let’s get the party started with the saved searches that handle the monitoring!
We utilize four saved searches to power all of our outage alerts – one for each type (index, host, sourcetype, source). The searches run every 10 minutes with a lookback of 9 days. You might notice that in our alert (below), we use isnull(indexed_delta), which helps us alert on any dataset that has not sent logs in for 9+ days, unless, of course, we exclude the monitoring for that data. This is the schedule and time window we picked based on our needs and data, however it is highly customizable. Our alerts post to a messaging platform and mention the SMEs directly, as well as page our team through Splunk On-Call (VictorOps), which lets us escalate to the data owner's team. The alerts also trigger an email alert action for any email address provided in the lookup. Every saved search contains a collect command to write the results to a summary index that we use in a separate dashboard to track outages over time and determine if a problem management ticket is necessary to fix recurring problems.
Pro tip: For alert messaging, make it informative and quick to read. We notify the SMEs about the data in question, add in the related runbook link, and show how long it's been since we last saw data compared to when we expected it. If you’re feeling up for it and it isn’t adding noise, add some personality to the messages, such as “Hey, thanks for checking in! We’re noticing an issue with ....”.
The basis of our search is as follows:
| tstats max(_indextime) as latest_occurrence values(host) as possibly_affected_hosts where [ | inputlookup index_monitoring.csv where NOT exclude=*| table index] by index | inputlookup append=t index_monitoring.csv where NOT exclude=* | stats values(*) as * by index | eval indexed_delta=now()-latest_occurrence | eval day=strftime(now(),"%a") | eval hour=strftime(now(),"%H") | eval current_epoch=now() | search [ | inputlookup index_monitoring.csv where exclude_time=* | table index exclude_time | appendcols [ | inputlookup bank_holidays.csv | where now()>=start AND now()<=end ] | filldown | eval exclude_time=if(like(exclude_time,"%Holiday%"),exclude_time." OR (current_epoch>=".start." AND current_epoch<=".end.")",exclude_time) | table index exclude_time | eval index="index=".index | eval exclude_time="(". exclude_time.")" | return 1000 $$index $$exclude_time | eval search="NOT (".search.")" | eval search=if(search="NOT ()","*",search)] | where frequency_s<indexed_delta OR isnull(indexed_delta) | fields index possibly_affected_hosts latest_occurrence frequency_s indexed_delta sme owner paging critical email runbook | eval latest_occurrence_epoch=latest_occurrence | eval possibly_affected_hosts=mvjoin(possibly_affected_hosts,",")
Pro tip: If you have metrics indexes, append an mstats command after the tstats command so it returns the same latest_occurrence and possibly_affected_hosts fields.
Note: we learned we needed to use $$<field> in the return statement of the subsearch when we created this as an alert. When just running the SPL alone, only one $ is necessary, but once scheduled, it requires two $.
The sourcetype-level search follows the same base, with a few tweaks: it aggregates on index and sourcetype together using the sourcetype_monitoring.csv lookup as a filter. It also runs a second lookup command against index_monitoring.csv afterward to fill in the empty fields from the sourcetype-level lookup. Host- and source-level searches follow this same template, just swapping in their respective lookups and metadata fields.
We use the following settings on our alerts to make sure they are suppressing alerts for 2 hours by the interesting fields (index, sourcetype, source, host), to give engineers a chance to resolve the issues, as well as ensuring we are alerting per result.
alert.digest_mode = 0 alert.suppress = 1 alert.suppress.fields = <metadata fields> alert.suppress.period = 2h
When an alert fires, we open a Splunk dashboard built for troubleshooting that specific outage. It links to the runbook, pulls in the owner and threshold details from the lookups, and lists the hosts sending the data. Under the hood it's tstats searches with tokens for the index, source, sourcetype, or host, combined with those same lookups.
In the example photo above, we can see that the audit sourcetype from the active_directory index went into outage when it spiked above the orange threshold line. The purple line indicates how long it’s been since we got new data – the lower the number, the more frequent data is brought in, which is why the login sourcetype looks like a flat line at the bottom. The next two panels show us timecharts of the timestamps we are getting events in (this helps us determine if data is backfilling) as well as the event counts (which helps us understand if we’re seeing any major spike before or after the outage). Finally, we have the keyword search at the bottom that looks over all error logs. This sometimes results in a red herring, but it is very useful when trying to find any issues.
So, What Does This Actually Look Like?
Let's walk through a typical alert flow. At 2:47 am, an alert fires: index_b hasn't seen new data in 14 minutes, well past its 10-minute expected frequency. The alert pages our team, sends a message tagging the SMEs, and links straight to the related runbook.
From there, the on-call engineer opens the data outage troubleshooting dashboard, which does most of the talking. The _indextime timechart shows a clean drop at 2:33 am, so we know exactly when things broke. The eventcount panel shows no gradual decline beforehand, which rules out a slow leak. A quick keyword search using internal logs turns up the answer: a certificate expired on the forwarder handling that data.
From alert to root cause: about six minutes, most of it spent just reading the dashboard. This is a critical index, so we page the SME directly through the on-call tool. The SME rotates the certificate, data starts flowing again by 3:10 am, and the summary index quietly logs the whole thing for us to review later. There's an old saying we've used for incidents this easy: 'two clicks and a beer.' We'll call this one 'two clicks and a coffee,' because we got the cert rotated and data flowing, all before the coffee even got cold.
The Upkeep
The downside of this specific solution is that it requires that any updates to the lookups are completed manually. These updates could include updating existing rows of the lookup or adding new rows. When adding new indexes (and other metadata), we follow a similar process to the initial deployment, by analyzing how frequently we expect data to come in and tracking that over a period of time, before permanently turning on the monitoring. Our team manages all these files through our version control platform and deploys the changes through automation pipelines. When a change is needed, a merge request is opened, approved, merged, and deployed.
There are ways in which this process can be improved, however, such as a web form for end-users to submit updates that automatically create the merge request. We, as a team, prefer these files to be managed in version control. If a breaking change is accidentally deployed, we can easily roll it back to the latest working version.
We do have additional alerting in place to inform us if any of our listed contacts are no longer with the company or potentially moved teams, indicating that we will need to update all references within our lookups for that employee. We also don't rely on memory to catch gaps. Separate alerts run daily to flag any index, host, sourcetype, or source that's actively sending data but missing from its respective lookup. This ensures that if someone spins up a new data source and forgets to tell us (or we forget to add it to the lookups), we are notified and can begin monitoring.
The search we use to find missing monitors essentially pulls a list of all indexes deployed using a rest search and compares it to the index monitoring lookup. Similarly, for sourcetypes, hosts, and sources, we compare the monitoring lookup to a list of all sourcetypes, hosts, or sources coming in using a tstats command filtered on the lookup. Here is an example of the SPL we use to check all sourcetypes are accounted for in the lookup:
| tstats count as _count where [ | inputlookup sourcetype_monitoring.csv | stats count as _count by index] by index sourcetype | lookup sourcetype_monitoring.csv index sourcetype OUTPUT index as st_monitoring | where isnull(st_monitoring) | fields - st_monitoring
The Results
We can’t be all talk and no walk!
As it stands today, we have a total of
334 indexes defined in our base index lookup
88 indexes actively being monitored
6 indexes temporarily disabled from monitoring
240 not monitored at the index level: either monitored by sourcetype, host, or source instead (23 indexes across 184 sourcetypes, 30 across 79 hosts, 7 across 27 sources), or excluded because they're unused, summary, or decommissioned.
Note: We do not monitor some of the out-of-the-box indexes that don’t receive data frequently enough, nor do we monitor summary indexes, since we would catch issues at the source data level. We have other alerts that monitor for saved search issues that would catch specific problems with summary index generating searches. We also keep all existing indexes in our lookup for reference in case we need to know who owned that data or any details about it.
Over a recent 30 day period, we have caught 4 total index outages, 23 host outages, 8 sourcetype outages, and 1 source outage. These alerts have resulted in faster time to detect and remediate data outages, ensuring a more resilient environment for our IT and Security teams.
Final Thoughts
While this solution involves significant setup, manual upkeep, and static thresholds, it is a very dependable solution once tuned properly. We had originally tried to use machine learning to help detect anomalies many years ago, however it was too resource-heavy due to the amount of data it had to analyze. It may be a solution we attempt again with Splunk becoming more and more efficient and with new product features becoming more and more powerful. We are also looking into using Splunk SOAR and other methods to provide details to the engineer upon the alert triggering as well as to remediate specific issues without involving an engineer at all
If your team has tackled data-outage monitoring differently, I'd love to hear about it. Please drop a comment below!
Credit to my amazing team and all the amazing work they do: @jewnix-splunk , @durrilk_splunk , @m_pham, @asoares , @ghuang-splunk Want updates like this sent straight to you? Learn how to subscribe to this blog (and follow Labels you care about) in our quick guide.
... View more