topic Re: How to write a search to alert if indexers are not receiving data from forwarders? in Splunk Search

How to write a search to alert if indexers are not receiving data from forwarders?

srikanth1213 — Mon, 10 Oct 2016 15:52:50 GMT

Hi Team,

How do I write a search to alert me when one of the critical indexers is not receiving the data from the source?

Re: How to write a search to alert if indexers are not receiving data from forwarders?

somesoni2 — Mon, 10 Oct 2016 21:31:36 GMT

You can use following query to check if you're receiving data from a particular source OR not.

Assuming your threshold time period for not reporting is 1 hour/3600 secs. Run below search for a period longer then 1 hr and setup alert when there are records retured

Using metadata command
If you want to know based on host

| metadata type=hosts index=yourindexNameHere | where host=yourHostNameHere| eval age=(recentTime-now()) | where age>3600 | table host recentTime age | convert ctime(recentTime)

For sourcetype, use

| metadata type=sourcetypes index=yourindexNameHere | where sourcetype=yourSourcetypeNameHere| eval age=(recentTime-now()) | where age>3600 | table sourcetype recentTime age | convert ctime(recentTime)

For source, use

| metadata type=sources index=yourindexNameHere | where source=yourSourceNameHere| eval age=(recentTime-now()) | where age>3600 | table source recentTime age | convert ctime(recentTime)

Using tstats
Just replace sourcetype with any other metadata field that you want to use.

| tstats max(_time) as recentTime WHERE index=yourindexNameHere  by sourcetype | where sourcetype=yourSourcetypeNameHere| eval age=(recentTime-now()) | where age>3600 | table sourcetype recentTime age | convert ctime(recentTime)

Re: How to write a search to alert if indexers are not receiving data from forwarders?

srikanth1213 — Fri, 14 Oct 2016 17:36:49 GMT

@somesoni2 :thanks much for the answer. I should have put the question in the right way ..my bad !
Basically I wanted to check if critical "indexes" but not "indexer" are receiving the data or not..
Does the above answer's apply for indexes as well?

Re: How to write a search to alert if indexers are not receiving data from forwarders?

somesoni2 — Fri, 14 Oct 2016 18:12:53 GMT

It actually applies to indexes only. If you've setup your indexers in standard way, all indexers are important. What you want is very common Splunk monitoring use-case. Hope the answer helps.

Re: How to write a search to alert if indexers are not receiving data from forwarders?

skoelpin — Fri, 14 Oct 2016 18:23:09 GMT

If your on v6.4 you could use the DMC to monitor your forwarders, if on an earlier version then you can do something like this

index=_internal sourcetype=splunkd destPort!="-"| stats sparkline count by hostname, sourceHost, host, destPort, version | rename destPort as "Destination Port" | rename host as "Indexer" | rename sourceHost as "Forwarder IP" | rename version as "Splunk Forwarder Version" | rename hostname as "Forwarder Host Name" | rename sparkline as "Traffic Frequency" | sort - count

Re: How to write a search to alert if indexers are not receiving data from forwarders?

srikanth1213 — Mon, 17 Oct 2016 17:06:07 GMT

Hi somesoni2, in your statement "Assuming your threshold time period for not reporting is 1 hour/3600 secs. Run below search for a period longer then 1 hr and setup alert when there are records retured" did you mean " setup alert when there are records "NOT "returned ?