Alerting

How to set up an alert to trigger if certain indexes do not receive any events within a certain time range?

kestasm
Path Finder

Hello,

We are monitoring a few indexes which are very important for us, but sometimes services get stuck and events are not sent to the indexes. Is it possible to create an alert which would send us a notification when there is no event condition met in an hour or any similar timeline?

I tried to using this, but it seems it doesn't work:

|metadata type=index |search index=A OR index=B OR index=C |eval age=now() - lastTime | search age>3600 
Tags (3)
1 Solution

aaronkorn
Splunk Employee
Splunk Employee

Try this and adjust your thresholds accordingly then create a search that alerts when it matches your alert criteria:

| tstats latest(_time) AS lastTime WHERE index=_internal OR index=_audit BY index | eval current=now() | eval age_min=round((current-lastTime)/60,2) | rangemap field=age_min default=severe low=0-30 elevated=30-60 | stats max(current) AS "Current Time" max(lastTime) AS "Latest Event" by range index age_min | convert ctime(*Time) ctime("Latest Event")  | rename  age_min AS "Communicated Minutes Ago"

View solution in original post

aaronkorn
Splunk Employee
Splunk Employee

Try this and adjust your thresholds accordingly then create a search that alerts when it matches your alert criteria:

| tstats latest(_time) AS lastTime WHERE index=_internal OR index=_audit BY index | eval current=now() | eval age_min=round((current-lastTime)/60,2) | rangemap field=age_min default=severe low=0-30 elevated=30-60 | stats max(current) AS "Current Time" max(lastTime) AS "Latest Event" by range index age_min | convert ctime(*Time) ctime("Latest Event")  | rename  age_min AS "Communicated Minutes Ago"

aaronkorn
Splunk Employee
Splunk Employee

Assuming you're on Splunk 6.x, something like the following should do the trick. Also, do you care about the index as a whole or the different sourcetypes within it?

| tstats latest(_time) AS lastTime WHERE index=A OR index=B OR index=C  BY host sourcetype index | eval current=now() | eval age_min=round((current-lastTime)/60,2) | rangemap field=age_min default=severe low=0-30 elevated=30-60 | stats max(current) AS "Current Time" values(sourcetype) AS Sourcetypes list(lastTime) AS "Latest Event" list(age_min) AS age_min list(range) AS Status by host | convert ctime(*Time) ctime("Latest Event")  | eventstats avg(age_min) AS avg_age by host | eval avg_age=round(avg_age,2) | sort - avg_age | rename  age_min AS "Communicated Minutes Ago" avg_age AS "Avg Communicated Minutes Ago"
0 Karma

Pawlub1
Engager

I like the search, but what are we using as the trigger.

Typically, something like 'search count < 1' for a zero trigger

I tried 'Communicated Minutes Ago < 1' ,but that is not triggering the alert. 

Cheers, Paul

Tags (1)
0 Karma

kestasm
Path Finder

Just the index as a whole.

0 Karma
Get Updates on the Splunk Community!

Adoption of RUM and APM at Splunk

    Unleash the power of Splunk Observability   Watch Now In this can't miss Tech Talk! The Splunk Growth ...

March Community Office Hours Security Series Uncovered!

Hello Splunk Community! In March, Splunk Community Office Hours spotlighted our fabulous Splunk Threat ...

Stay Connected: Your Guide to April Tech Talks, Office Hours, and Webinars!

Take a look below to explore our upcoming Community Office Hours, Tech Talks, and Webinars in April. This post ...