topic Alert if data not received to an index for 1 hour in Splunk Dev

Alert if data not received to an index for 1 hour

denose — Wed, 14 Mar 2018 22:51:27 GMT

I am currently using this search:

index=_audit OR index=_internal OR index=_introspection OR index=a OR index=b OR index=c OR index=d earliest=-1h latest=-59m | stats count by index | where count=0

I tried input lookups for listing all the indexes but didn't get far quickly. If I take off the where command, it will show a count of events for only the indexes that have events. So when I say where=0 there are not indexes to be seen.

How can I make it search those indexes and tell me if they have in fact had no events?

Re: Alert if data not received to an index for 1 hour

skoelpin — Wed, 14 Mar 2018 23:56:02 GMT

This wont work since your looking for a count equal to zero rather than null values. You may also want to change your time fields since your alerts wlll arrive an hour late

Try this

| makeresults | eval count="" 
| append [ | search index=_audit OR index=_internal OR index=_introspection OR index=a OR index=b OR index=c OR index=d earliest=-1h latest=-59m | stats count by index | fillnull value=0 | where count=0]

Re: Alert if data not received to an index for 1 hour

denose — Thu, 15 Mar 2018 02:41:41 GMT

Thanks, I still don't get any indexes listed with 0 events.
It just returns, under the Statistics tab, the current timestamp in the _time column and blank under count column?

Re: Alert if data not received to an index for 1 hour

mayurr98 — Thu, 15 Mar 2018 07:28:47 GMT

You can use following query to check if you're receiving data from a particular source OR not.

As your threshold time period for not reporting is 1 hour/3600 secs. Run below search for a period longer than 1 hr and set up an alert when there are records returned

Using metadata command
If you want to know based on host

| metadata type=hosts index=yourindexNameHere | where host=yourHostNameHere| eval age=(recentTime-now()) | where age>3600 | table host recentTime age | convert ctime(recentTime)

For sourcetype, use

| metadata type=sourcetypes index=yourindexNameHere | where sourcetype=yourSourcetypeNameHere| eval age=(recentTime-now()) | where age>3600 | table sourcetype recentTime age | convert ctime(recentTime)

For source, use

| metadata type=sources index=yourindexNameHere | where source=yourSourceNameHere| eval age=(recentTime-now()) | where age>3600 | table source recentTime age | convert ctime(recentTime)

Using tstats
Just replace sourcetype with any other metadata field that you want to use.

| tstats max(_time) as recentTime WHERE index=yourindexNameHere  by sourcetype | where sourcetype=yourSourcetypeNameHere| eval age=(recentTime-now()) | where age>3600 | table sourcetype recentTime age | convert ctime(recentTime)

let me know if this helps!

Re: Alert if data not received to an index for 1 hour

denose — Thu, 15 Mar 2018 22:32:52 GMT

Thanks mayurr98 though, I'm not sure how this works on a per index basis?
I want to know which index hasn't received events rather than source, sourcetype or hosts.

Re: Alert if data not received to an index for 1 hour

mayurr98 — Fri, 16 Mar 2018 06:58:23 GMT

Try this

| tstats max(_time) as recentTime WHERE index=*  by index | eval age=now()-recentTime| where age>3600 | table index recentTime age | convert ctime(recentTime)

So what I have done is per index I took recent event time and then subtracted it from the current time to calculate age and then applied a condition on age to get the index details. Run this for all time to get accurate results.

Re: Alert if data not received to an index for 1 hour

denose — Mon, 19 Mar 2018 04:28:37 GMT

Thanks for that. I think that's enough to do what I wanted for now.

I appreciate your assistance.