We have thousands of forwarders installed. Our requirement is to check status of only 50-100 forwarders from thousands of forwarders on daily basis.
These selective 50-100 forwarders will be new every time. We will use CSV file to give forwarders input to search query.
Below search query we are using,
index=internal sourcetype=splunkd group=tcpinconnections (connectionType=cooked OR connectionType=cookedSSL) fwdType=* guid=* [| inputlookup forwarders.csv | fields hostname]
| stats values(fwdType) as forwardertype, latest(version) as version, values(arch) as arch, values(os) as os, max(time) as lastconnected by guid, hostname
| eval status = if(lastconnected < (now() - 900), "missing", "active")
| eval lastconnected=strftime(lastconnected,"%m/%d/%y %H:%M:%S")
| fields guid, hostname, forwardertype, version, arch, os, status, lastconnected
This query is working as per the expectations. However I have one more concern i.e.
We want to see the status of forwarders as "Not Found", For those forwarders which does not exist in enviornment or For those forwarders which log entries are not present in internal index.
This query is not returning status for those forwarders which does not exist or whose log entries are not available.
I will be thankful If any of you can guide or provide search query.
MetaWoot combined with a custom query to determine when the forwarders last sent data in would probably solve your issue...