Alerting
Highlighted

How to set up an alert to trigger if certain indexes do not receive any events within a certain time range?

Path Finder

Hello,

We are monitoring a few indexes which are very important for us, but sometimes services get stuck and events are not sent to the indexes. Is it possible to create an alert which would send us a notification when there is no event condition met in an hour or any similar timeline?

I tried to using this, but it seems it doesn't work:

|metadata type=index |search index=A OR index=B OR index=C |eval age=now() - lastTime | search age>3600 
Tags (3)
Highlighted

Re: How to set up an alert to trigger if certain indexes do not receive any events within a certain time range?

Splunk Employee
Splunk Employee

Assuming you're on Splunk 6.x, something like the following should do the trick. Also, do you care about the index as a whole or the different sourcetypes within it?

| tstats latest(_time) AS lastTime WHERE index=A OR index=B OR index=C  BY host sourcetype index | eval current=now() | eval age_min=round((current-lastTime)/60,2) | rangemap field=age_min default=severe low=0-30 elevated=30-60 | stats max(current) AS "Current Time" values(sourcetype) AS Sourcetypes list(lastTime) AS "Latest Event" list(age_min) AS age_min list(range) AS Status by host | convert ctime(*Time) ctime("Latest Event")  | eventstats avg(age_min) AS avg_age by host | eval avg_age=round(avg_age,2) | sort - avg_age | rename  age_min AS "Communicated Minutes Ago" avg_age AS "Avg Communicated Minutes Ago"
0 Karma
Highlighted

Re: How to set up an alert to trigger if certain indexes do not receive any events within a certain time range?

Path Finder

Just the index as a whole.

0 Karma
Highlighted

Re: How to set up an alert to trigger if certain indexes do not receive any events within a certain time range?

Splunk Employee
Splunk Employee

Try this and adjust your thresholds accordingly then create a search that alerts when it matches your alert criteria:

| tstats latest(_time) AS lastTime WHERE index=_internal OR index=_audit BY index | eval current=now() | eval age_min=round((current-lastTime)/60,2) | rangemap field=age_min default=severe low=0-30 elevated=30-60 | stats max(current) AS "Current Time" max(lastTime) AS "Latest Event" by range index age_min | convert ctime(*Time) ctime("Latest Event")  | rename  age_min AS "Communicated Minutes Ago"

View solution in original post