What's the best way to determine that a forwarder is connected to an indexer? I don't want to base it on the last time data was seen, as some of these are low volume and sporadic (but still important) feeds. Is there a way to test for each forwarder that was connected at least once, and run a search to test that that is now connected? Is there a heartbeat or check-in setting and/or messsage?
You can use the following -
| inputlookup <host list> | fields host | join type=left host [| metadata type=hosts index=* | eval host=lower(host) | eval _time=recentTime | sort host, _time | stats latest(_time) as recentTime by host ] | eval LAST=strftime(recentTime,"%a %m/%d/%Y-%T %Z(%z)"), DAYS_AGO=round((recentTime-now())/86400,0)
Where in the lookup you have the field
host. You sort then by
It would look like the following with the list of hosts -
host!=forwarder in many architectures, especially when collecting events from many hosts through a syslog server for instance, the host field will hold the name/ip of the original host, not that of the syslog server / forwarder.
And perhaps you intended to filter that using that lookup with the host list, but then still that will not work well given that it is stated in the question that the data sources can be very infrequent in some cases. And index=* will ignore internal indexes.
Forwarders also send their internal logs, which, even when the actual data inputs are very sparse, should still occur with a suitable frequency to establish whether they are still connected. But I guess the most straighforward way is looking at the metrics.log from your indexer, focusing on the tcpin_connections group:
index=_internal group="tcpin_connections" | stats latest(_time) by sourceHost
Which will give you the most recent connection from each forwarder.