Please share SPL to help in Finding Forwarders that are Broken / Not calling home any more. Is it possible to view this in GUI? Thank u
Sir thank as always for your accurate responses. I have one question. Reg. Missing FWs , I have Apps Meta Woot & Splunk Admins, I can get a list of missing FWs no problem. I have found that some of the missing are due to the server they reside on are decommissioned, find. But of the others I can not figure out if the FW is broken / needs updating to function. Please advise. Thank u as always.
I’m afraid that there is not an easy solution for this. Probably best way is that you have boarding/integrating process where you are collecting the needed information from source systems. Or if you have any CMDB then you could utilize it to ask from owners of those systems what is the situation if you haven’t any other automated way to check those.
This is actually something I'm currently working on, and you're correct when you say it isnt easy. I had to create a custom alert action which when my missing forwarder alert is triggered, will execute a python script to first ping the host to see if its available, and then also checks the hosts operation status from our CMDB.
Thank u sir very much & good to hear from you. I missed learning from you for a short while. Stay safe.
We use a lookup of all hosts reporting to splunk, and do a join with metadata for current hosts and most recent event. It allows us to see the hosts that havent reported in a day.
| inputlookup your_host_lookup
| eval host = lower(host)
| join host type=outer
[| metadata type=hosts index=_internal
| eval host = lower(host)
| sort - lastTime
| dedup host
| eval minutesSinceLastEvent = ( ( now() - recentTime ) / 60 )
| eval daysSinceLastEvent = ( minutesSinceLastEvent / 1440 )]
| where daysSinceLastEvent >= 1
| eval lastEventDays=round(daysSinceLastEvent,2)
| eval lastEventMins=round(minutesSinceLastEvent,0)
Thank u for your response. So I need to have a complete list of all hosts in a .CSV file right? Where in Splunk do I place this file for the search to find it please? Thank u again
We use the Splunk kvstore. You can just have a search that runs daily to outputlookup your hosts list though. Something like this...
index=_internal | stats count by host | fields - count | outputlookup your_hosts.csv