Alerting

What is the best and most efficient way to write alert for index with no events?

dannyze
Explorer

What is the best and most efficient way to write alert for index with no events?

I have the following

index=_internal earliest=60m | where count=0

or

| metadata type=sources index=* | eval flatline=round((now()-recentTime)/60,0)

Thank You

Tags (1)
0 Karma
1 Solution

manjunathmeti
SplunkTrust
SplunkTrust

To find indexes with no events use both eventcount and tstats.

| eventcount summarize=false index=* | fields index | dedup index | join type=left [ | tstats count as event_count WHERE (index=* earliest=-60m) by index] | fillnull value=0 | where event_count=0

View solution in original post

woodcock
Esteemed Legend

This has been solved many times including:

Meta Woot!: https://splunkbase.splunk.com/app/2949/
TrackMe: https://splunkbase.splunk.com/app/4621/,
Broken Hosts App for Splunk: https://splunkbase.splunk.com/app/3247/
Alerts for Splunk Admins ("ForwarderLevel" alerts): https://splunkbase.splunk.com/app/3796/
Splunk Security Essentials(https://docs.splunksecurityessentials.com/features/sse_data_availability/): https://splunkbase.splunk.com/app/3435/
Monitoring Console: https://docs.splunk.com/Documentation/Splunk/latest/DMC/Configureforwardermonitoring
Deployment Server: https://docs.splunk.com/Documentation/DepMon/latest/DeployDepMon/Troubleshootyourdeployment#Forwarde...
0 Karma

dannyze
Explorer

Thank you, was looking more for a way to do it with built-in capabilities

0 Karma

woodcock
Esteemed Legend

Most of these are built-in capabilities (searches) with some gift-wrapping around that. My point is: don't reinvent the wheel: download some of these apps (or in the case of #6 and #7, just turn them on) and tear apart their searches and copy what you need.

gjanders
SplunkTrust
SplunkTrust

If you would prefer to go down the apps path
TrackMe
Meta Woot!
Broken Hosts App for Splunk

Or see previous answers for missing indexes/sourcetypes such as this one

Alerts for Splunk Admins https://splunkbase.splunk.com/app/3796/
Version Control for Splunk https://splunkbase.splunk.com/app/4355/
0 Karma

niketnilay
Legend

@dannyze if you want a different approach with REST API you can try the following however commands like tstats, metadata, eventcount and dbinspect are specifically useful while trying to query index related stuff. Since the following approach uses REST API you can output results similar to how you see it in Settings> Data > Indexes view (bring in or filter based on other fields like app name, access etc.

| rest /servicesNS/-/-/data/indexes 
| fields title maxTime
| rename title as index
| eval _time=strptime(maxTime,"%Y-%m-%dT%H:%M:%S+%z"), "Last event indexed age"=now()-_time 
| where 'Last event indexed age'>=3600 OR isnull('Last event indexed age')
| eval "Last event indexed age"=if(isnull('Last event indexed age'),"No Data",
                                   replace(replace(tostring('Last event indexed age',"duration"),"\+"," days "),"(\d+)\:(\d+)\:(\d+)\.\d+","\1 hr \2 min \3 sec"))
| fields - maxTime _time
____________________________________________
| makeresults | eval message= "Happy Splunking!!!"
0 Karma

to4kawa
Ultra Champion
| tstats count where index=* earliest=-60m by index 
| append [| eventcount summarize=f index=* |fields index]
| fillnull count
| where count=0

To fire alert: event count > 0
hi, @manjunathmeti

I modify your query.

0 Karma

manjunathmeti
SplunkTrust
SplunkTrust

To find indexes with no events use both eventcount and tstats.

| eventcount summarize=false index=* | fields index | dedup index | join type=left [ | tstats count as event_count WHERE (index=* earliest=-60m) by index] | fillnull value=0 | where event_count=0

View solution in original post

Did you miss .conf21 Virtual?

Good news! The event's keynotes and many of its breakout sessions are now available online, and still totally FREE!