How to alert when the last event in an index is older than 1 hour, and if any forwarder has not phoned home in the last hour?

[guimilare]

Hello Splunkers.

I'm working on 2 scenarios, and I'm sure you guys can help me.

Scenario 1:
We have about 15 indexes.
Sometimes, the applications that generate data to Splunk gets stuck and no data is indexed.
What I need is to alert the user every time the last event indexed in an index is older than 1 hour.
Using metadata I'm able to deal with only 1 index at a time.

Scenario 2:
We have now about 50 forwarders sending data to Splunk.
What is the best way to alert the user if any forwarder did not phoned home in the last hour?

Thanks in advance.
Regards.

[gcusello]

Hi guimilare,
this method is applicable both your scenarios.
You have to create a lookup with all the indexes/forwarders to monitor (e.g.: indexes.csv or perimeter.csv) and then run a search like these using as time period one hour:

| inputlookup indexes.csv | eval count=0 | append [ index=* | stats count by index ] | stats sum(count) AS total by index | where total=0

(instead to use index=* you could create a tag or an eventtype with all your index to monitor)

| inputlookup perimeter.csv | eval count=0, host=lower(host) | append [ index=_internal | eval host=lower(host)| stats count by index ] | stats sum(count) AS total by index | where total=0

In this way you have indexes and forwarders that doesn't send logs in the selected period.

If you like, instead to have condition |where total=0 you could use rangemap command and have a visualization of the indexes or forwarders status (also graphically).

Bye.
Giuseppe

[skoelpin]

What version are you running?

If >= 6.4 then you could use the DMC to do this

Or you could set up an alert with this search

index=_internal sourcetype=splunkd destPort!="-"| stats sparkline count by hostname, sourceHost, host, destPort, version | rename destPort as "Destination Port" | rename host as "Indexer" | rename sourceHost as "Forwarder IP" | rename version as "Splunk Forwarder Version" | rename hostname as "Forwarder Host Name" | rename sparkline as "Traffic Frequency" | sort - count