topic Re: Please share SPL to help in Finding Forwarders that are Broken / Not calling home any more. Thank u in Reporting

Please share SPL to help in Finding Forwarders that are Broken / Not calling home any more. Thank u

SamHTexas — Tue, 15 Jun 2021 20:52:52 GMT

Please share SPL to help in Finding Forwarders that are Broken / Not calling home any more. Is it possible to view this in GUI? Thank u

Re: Please share SPL to help in Finding Forwarders that are Broken / Not calling home any more. Thank u

pbarbuto — Wed, 16 Jun 2021 13:59:08 GMT

We use a lookup of all hosts reporting to splunk, and do a join with metadata for current hosts and most recent event. It allows us to see the hosts that havent reported in a day.

Re: Please share SPL to help in Finding Forwarders that are Broken / Not calling home any more. Thank u

SamHTexas — Wed, 16 Jun 2021 19:15:11 GMT

Thank u for your response. So I need to have a complete list of all hosts in a .CSV file right? Where in Splunk do I place this file for the search to find it please? Thank u again

Re: Please share SPL to help in Finding Forwarders that are Broken / Not calling home any more. Thank u

isoutamo — Wed, 16 Jun 2021 20:58:59 GMT

If/when you have MC on place, you could set up Forwarder monitoring there and even get automatic alerts. Of course this needs, that nobody will rebuild forwarders lookup before missing FWDs has fixed.
Another option is use eg. meta woot app for this.

On Splunk Usergroup Slack there is list of other options:
There are a lot of options for finding hosts or sources that stop submitting events:
Meta Woot! https://splunkbase.splunk.com/app/2949/
TrackMe https://splunkbase.splunk.com/app/4621/
Broken Hosts App for Splunk https://splunkbase.splunk.com/app/3247/
Alerts for Splunk Admins ("ForwarderLevel" alerts) https://splunkbase.splunk.com/app/3796/
Monitoring Console https://docs.splunk.com/Documentation/Splunk/latest/DMC/Configureforwardermonitoring
Deployment Server https://docs.splunk.com/Documentation/DepMon/latest/DeployDepMon/Troubleshootyourdeployment#Forwarder_warnings
Some helpful posts:
https://lantern.splunk.com/hc/en-us/articles/360048503294-Hosts-logging-data-in-a-certain-timeframe
https://www.duanewaddle.com/proving-a-negative/
r. Ismo

Re: Please share SPL to help in Finding Forwarders that are Broken / Not calling home any more. Thank u

pbarbuto — Wed, 16 Jun 2021 23:34:29 GMT

We use the Splunk kvstore. You can just have a search that runs daily to outputlookup your hosts list though. Something like this...

index=_internal | stats count by host | fields - count | outputlookup your_hosts.csv

Re: Please share SPL to help in Finding Forwarders that are Broken / Not calling home any more. Thank u

SamHTexas — Thu, 17 Jun 2021 02:06:38 GMT

Thank u sir very much & good to hear from you. I missed learning from you for a short while. Stay safe.

Re: Please share SPL to help in Finding Forwarders that are Broken / Not calling home any more. Thank u

SamHTexas — Thu, 17 Jun 2021 14:05:58 GMT

Sir thank as always for your accurate responses. I have one question. Reg. Missing FWs , I have Apps Meta Woot & Splunk Admins, I can get a list of missing FWs no problem. I have found that some of the missing are due to the server they reside on are decommissioned, find. But of the others I can not figure out if the FW is broken / needs updating to function. Please advise. Thank u as always.

Re: Please share SPL to help in Finding Forwarders that are Broken / Not calling home any more. Thank u

isoutamo — Tue, 29 Jun 2021 20:07:34 GMT

I’m afraid that there is not an easy solution for this. Probably best way is that you have boarding/integrating process where you are collecting the needed information from source systems. Or if you have any CMDB then you could utilize it to ask from owners of those systems what is the situation if you haven’t any other automated way to check those.

Re: Please share SPL to help in Finding Forwarders that are Broken / Not calling home any more. Thank u

pbarbuto — Tue, 29 Jun 2021 20:14:22 GMT

This is actually something I'm currently working on, and you're correct when you say it isnt easy. I had to create a custom alert action which when my missing forwarder alert is triggered, will execute a python script to first ping the host to see if its available, and then also checks the hosts operation status from our CMDB.