topic Re: What is the best and most efficient way to write alert for index with no events? in Alerting

What is the best and most efficient way to write alert for index with no events?

dannyze — Thu, 20 Feb 2020 07:11:55 GMT

What is the best and most efficient way to write alert for index with no events?

I have the following

index=_internal earliest=60m | where count=0

| metadata type=sources index=* | eval flatline=round((now()-recentTime)/60,0)

Thank You

Re: What is the best and most efficient way to write alert for index with no events?

manjunathmeti — Thu, 20 Feb 2020 07:46:21 GMT

To find indexes with no events use both eventcount and tstats.

| eventcount summarize=false index=* | fields index | dedup index | join type=left [ | tstats count as event_count WHERE (index=* earliest=-60m) by index] | fillnull value=0 | where event_count=0

Re: What is the best and most efficient way to write alert for index with no events?

to4kawa — Thu, 20 Feb 2020 09:08:37 GMT

| tstats count where index=* earliest=-60m by index 
| append [| eventcount summarize=f index=* |fields index]
| fillnull count
| where count=0

To fire alert: event count > 0
hi, @manjunathmeti

I modify your query.

Re: What is the best and most efficient way to write alert for index with no events?

niketn — Thu, 20 Feb 2020 10:50:14 GMT

@dannyze if you want a different approach with REST API you can try the following however commands like tstats, metadata, eventcount and dbinspect are specifically useful while trying to query index related stuff. Since the following approach uses REST API you can output results similar to how you see it in Settings> Data > Indexes view (bring in or filter based on other fields like app name, access etc.

| rest /servicesNS/-/-/data/indexes 
| fields title maxTime
| rename title as index
| eval _time=strptime(maxTime,"%Y-%m-%dT%H:%M:%S+%z"), "Last event indexed age"=now()-_time 
| where 'Last event indexed age'>=3600 OR isnull('Last event indexed age')
| eval "Last event indexed age"=if(isnull('Last event indexed age'),"No Data",
                                   replace(replace(tostring('Last event indexed age',"duration"),"\+"," days "),"(\d+)\:(\d+)\:(\d+)\.\d+","\1 hr \2 min \3 sec"))
| fields - maxTime _time

Re: What is the best and most efficient way to write alert for index with no events?

gjanders — Sun, 23 Feb 2020 09:21:13 GMT

If you would prefer to go down the apps path
TrackMe
Meta Woot!
Broken Hosts App for Splunk

Or see previous answers for missing indexes/sourcetypes such as this one

Re: What is the best and most efficient way to write alert for index with no events?

woodcock — Sun, 23 Feb 2020 20:20:33 GMT

This has been solved many times including:

Meta Woot!: https://splunkbase.splunk.com/app/2949/
TrackMe: https://splunkbase.splunk.com/app/4621/,
Broken Hosts App for Splunk: https://splunkbase.splunk.com/app/3247/
Alerts for Splunk Admins ("ForwarderLevel" alerts): https://splunkbase.splunk.com/app/3796/
Splunk Security Essentials(https://docs.splunksecurityessentials.com/features/sse_data_availability/): https://splunkbase.splunk.com/app/3435/
Monitoring Console: https://docs.splunk.com/Documentation/Splunk/latest/DMC/Configureforwardermonitoring
Deployment Server: https://docs.splunk.com/Documentation/DepMon/latest/DeployDepMon/Troubleshootyourdeployment#Forwarder_warnings

Re: What is the best and most efficient way to write alert for index with no events?

dannyze — Sun, 23 Feb 2020 23:01:18 GMT

Thank you, was looking more for a way to do it with built-in capabilities

Re: What is the best and most efficient way to write alert for index with no events?

woodcock — Sun, 23 Feb 2020 23:06:51 GMT

Most of these are built-in capabilities (searches) with some gift-wrapping around that. My point is: don't reinvent the wheel: download some of these apps (or in the case of #6 and #7, just turn them on) and tear apart their searches and copy what you need.