Checking for forwaders if not reporting to indexers

[splunker9999]

Hi,

We have below search running to check if any forwarder is missing for the last 15 mins.
Here below we are running across index=*, if so we are getting an error message as below:

Error in 'metadata': No 'host' key found in results. Cannot merge metadata.

Incase If I run below query only for index=_internal, search is running good,inorder to check if forwader is missing do we need to check all indexes or _internal would be fine?

|metadata type=hosts index=*
| where (now()-recentTime<7200) 
| stats count by host recentTime]
| stats count by host recentTime
| rex field=host "(?P<hostname>[^.]+)" 
| eval hostname=lower(hostname)
| eval sourceHost=hostname
| eval connectionType="universal forwarder"
| eval arch="undefined"
| eval lastReceived = recentTime
| eval lastConnected=recentTime
| eval KB = round(1000, 4)
| eval eps = round(100, 4)
| eval os= case(like(sourceHost,"l%"), "Linux",like(sourceHost,"W%"), "Windows", like(sourceHost,"w%"), "Windows", like(sourceHost,"s%"), "Solaris", like(sourceHost,"S%"), "Solaris", 1=1,"Other")
| eval mystatus = if(lastConnected<(now()-100),"quiet","dunno")
| eval status = if(lastConnected<(now()-900),"missing",if(mystatus="quiet","quiet","active"))

Thanks

[woodcock]

Enable the DMC Alert - Missing Forwarders:

http://docs.splunk.com/Documentation/Splunk/6.4.2/DMC/Platformalerts

[skoelpin]

Try this

index=_internal sourcetype=splunkd destPort!="-"| stats sparkline count by hostname, sourceHost, host, destPort, version | rename destPort as "Destination Port" | rename host as "Indexer" | rename sourceHost as "Forwarder IP" | rename version as "Splunk Forwarder Version" | rename hostname as "Forwarder Host Name" | rename sparkline as "Traffic Frequency" | sort - count

Save the search as an alert and set the timeframe to 15 minutes, it will do exactly what your asking for

[splunker9999]

Hi,
If we use search with index=_internal does this gives us results for all forwarders or it gives us only that forward to internal indexes ?

If it gives only forwarders that reports to _internal ,what should be the case if we need to find for all other indexes?

Thanks

[skoelpin]

Did this query work for you?

[skoelpin]

It will return ALL forwarders regardless of what index they are in. Try running the search

[jkat54]

Assuming you forward your internal indexes, the _internal search should be fine.

Metadata isnt always what you want to use though... you might try searching just index=_internal for some of these things instead of using |metadata

[pradeepkumarg]

You don't need the index filter at all. It should just run fine with |metadata type=hosts

[splunker9999]

Yeah we tried using this, we are facing same error message when tried this.

 Error in 'metadata': No 'host' key found in results. Cannot merge metadata.