Splunk Search

How to write a search to alert if indexers are not receiving data from forwarders?

srikanth1213
Path Finder

Hi Team,

How do I write a search to alert me when one of the critical indexers is not receiving the data from the source?

skoelpin
SplunkTrust
SplunkTrust

If your on v6.4 you could use the DMC to monitor your forwarders, if on an earlier version then you can do something like this

index=_internal sourcetype=splunkd destPort!="-"| stats sparkline count by hostname, sourceHost, host, destPort, version | rename destPort as "Destination Port" | rename host as "Indexer" | rename sourceHost as "Forwarder IP" | rename version as "Splunk Forwarder Version" | rename hostname as "Forwarder Host Name" | rename sparkline as "Traffic Frequency" | sort - count
0 Karma

somesoni2
Revered Legend

You can use following query to check if you're receiving data from a particular source OR not.

Assuming your threshold time period for not reporting is 1 hour/3600 secs. Run below search for a period longer then 1 hr and setup alert when there are records retured

Using metadata command
If you want to know based on host

| metadata type=hosts index=yourindexNameHere | where host=yourHostNameHere| eval age=(recentTime-now()) | where age>3600 | table host recentTime age | convert ctime(recentTime)

For sourcetype, use

| metadata type=sourcetypes index=yourindexNameHere | where sourcetype=yourSourcetypeNameHere| eval age=(recentTime-now()) | where age>3600 | table sourcetype recentTime age | convert ctime(recentTime)

For source, use

| metadata type=sources index=yourindexNameHere | where source=yourSourceNameHere| eval age=(recentTime-now()) | where age>3600 | table source recentTime age | convert ctime(recentTime)

Using tstats
Just replace sourcetype with any other metadata field that you want to use.

| tstats max(_time) as recentTime WHERE index=yourindexNameHere  by sourcetype | where sourcetype=yourSourcetypeNameHere| eval age=(recentTime-now()) | where age>3600 | table sourcetype recentTime age | convert ctime(recentTime)

srikanth1213
Path Finder

@somesoni2 :thanks much for the answer. I should have put the question in the right way ..my bad !
Basically I wanted to check if critical "indexes" but not "indexer" are receiving the data or not..
Does the above answer's apply for indexes as well?

0 Karma

somesoni2
Revered Legend

It actually applies to indexes only. If you've setup your indexers in standard way, all indexers are important. What you want is very common Splunk monitoring use-case. Hope the answer helps.

0 Karma

srikanth1213
Path Finder

Hi somesoni2, in your statement "Assuming your threshold time period for not reporting is 1 hour/3600 secs. Run below search for a period longer then 1 hr and setup alert when there are records retured" did you mean " setup alert when there are records "NOT "returned ?

0 Karma
Get Updates on the Splunk Community!

Accelerate Service Onboarding, Decomposition, Troubleshooting - and more with ITSI’s ...

Accelerate Service Onboarding, Decomposition, Troubleshooting - and more! Faster Time to ValueManaging and ...

New Release | Splunk Enterprise 9.3

Hi Splunky people! We are excited to share the newest updates in Splunk Enterprise 9.3!Admins and Analyst can ...

2024 Splunk Career Impact Survey | Earn a $20 gift card for participating!

Hear ye, hear ye! The time has come again for Splunk's annual Career Impact Survey!  We need your help by ...