How to test that a forwarder is connected to an indexer?

[a212830]

Hi,

What's the best way to determine that a forwarder is connected to an indexer? I don't want to base it on the last time data was seen, as some of these are low volume and sporadic (but still important) feeds. Is there a way to test for each forwarder that was connected at least once, and run a search to test that that is now connected? Is there a heartbeat or check-in setting and/or messsage?

[ddrillic]

You can use the following -

| inputlookup <host list> 
| fields host 
| join type=left host 
    [| metadata type=hosts index=* 
    | eval host=lower(host) 
    | eval _time=recentTime 
    | sort host, _time 
    | stats latest(_time) as recentTime by host ] 
| eval LAST=strftime(recentTime,"%a %m/%d/%Y-%T %Z(%z)"), DAYS_AGO=round((recentTime-now())/86400,0)

Where in the lookup you have the field host. You sort then by DAYS_AGO.

It would look like the following with the list of hosts -

[FrankVl]

host!=forwarder in many architectures, especially when collecting events from many hosts through a syslog server for instance, the host field will hold the name/ip of the original host, not that of the syslog server / forwarder.

And perhaps you intended to filter that using that lookup with the host list, but then still that will not work well given that it is stated in the question that the data sources can be very infrequent in some cases. And index=* will ignore internal indexes.

[FrankVl]

Forwarders also send their internal logs, which, even when the actual data inputs are very sparse, should still occur with a suitable frequency to establish whether they are still connected. But I guess the most straighforward way is looking at the metrics.log from your indexer, focusing on the tcpin_connections group:

index=_internal group="tcpin_connections" | stats latest(_time) by sourceHost

Which will give you the most recent connection from each forwarder.

[ChrisG]

One way is to install the Splunk Deployment Monitor app, which shows you information about forwarder status over time and warnings for forwarders that appear to be missing.

[zsustar]

Oops!
This app is currently not available....

how to use CLI to test?