Splunk Enterprise Security

How to manage reports and alerts for 150+ indexes?


We have a ton of indexes and need to better understand which ones have stopped receiving events so that we can report and alert on them.

We have a Splunk Enterprise v7.3.3 distributed environment with multiple (non-clustered) indexers, and non-pooled search heads configured in standalone mode. Our DSV, SH, and ES are each individual hosts and our ES is configured as a secondary SH. We manage index changes via CLI edits of indexes.conf, a deployment app, and redeployment of server classes.

We currently use the below in a dashboard panel, which generates a list of all "0-count" indexes that haven't received events in over 24 hours, but as a static list, there's a lot of additional work to get a holistic view of what's changed and when. I'd prefer query logic over a new app, as we're already hoping to pare down some of (our own) 'bloat.'

## generates a list of all "0-count" indexes that haven't received events in over 24 hours...

|tstats count where (index=* earliest=-24h latest=now()) by index

|append [|inputlookup index_list.csv |eval count=0]

|stats max(count) as count by index

|where count=0

Thanks in advance!

0 Karma

Ultra Champion
 This has been solved many times including:
 Meta Woot!: https://splunkbase.splunk.com/app/2949/
 TrackMe: https://splunkbase.splunk.com/app/4621/,
 Broken Hosts App for Splunk: https://splunkbase.splunk.com/app/3247/
 Alerts for Splunk Admins ("ForwarderLevel" alerts): https://splunkbase.splunk.com/app/3796/
 Splunk Security Essentials(https://docs.splunksecurityessentials.com/features/sse_data_availability/): https://splunkbase.splunk.com/app/3435/
 Monitoring Console: https://docs.splunk.com/Documentation/Splunk/latest/DMC/Configureforwardermonitoring
 Deployment Server: https://docs.splunk.com/Documentation/DepMon/latest/DeployDepMon/Troubleshootyourdeployment#Forwarde...

From @woodcock recommend

Get Updates on the Splunk Community!

Enterprise Security Content Update (ESCU) | New Releases

In the last month, the Splunk Threat Research Team (STRT) has had 2 releases of new security content via the ...

Announcing the 1st Round Champion’s Tribute Winners of the Great Resilience Quest

We are happy to announce the 20 lucky questers who are selected to be the first round of Champion's Tribute ...

We’ve Got Education Validation!

Are you feeling it? All the career-boosting benefits of up-skilling with Splunk? It’s not just a feeling, it's ...