Reporting

Please share SPL to help in Finding Forwarders that are Broken / Not calling home any more. Thank u

SamHTexas
Builder

Please share SPL to help in Finding Forwarders that are Broken / Not calling home any more. Is it possible to view this in GUI? Thank u

Labels (1)
Tags (1)
0 Karma

isoutamo
SplunkTrust
SplunkTrust
If/when you have MC on place, you could set up Forwarder monitoring there and even get automatic alerts. Of course this needs, that nobody will rebuild forwarders lookup before missing FWDs has fixed.
Another option is use eg. meta woot app for this.

On Splunk Usergroup Slack there is list of other options:
There are a lot of options for finding hosts or sources that stop submitting events:
Meta Woot! https://splunkbase.splunk.com/app/2949/
TrackMe https://splunkbase.splunk.com/app/4621/
Broken Hosts App for Splunk https://splunkbase.splunk.com/app/3247/
Alerts for Splunk Admins ("ForwarderLevel" alerts) https://splunkbase.splunk.com/app/3796/
Monitoring Console https://docs.splunk.com/Documentation/Splunk/latest/DMC/Configureforwardermonitoring
Deployment Server https://docs.splunk.com/Documentation/DepMon/latest/DeployDepMon/Troubleshootyourdeployment#Forwarde...
Some helpful posts:
https://lantern.splunk.com/hc/en-us/articles/360048503294-Hosts-logging-data-in-a-certain-timeframe
https://www.duanewaddle.com/proving-a-negative/
r. Ismo

SamHTexas
Builder

Sir thank as always for your accurate responses. I have one question. Reg. Missing FWs , I have Apps Meta Woot & Splunk Admins, I can get a list of missing FWs no problem. I have found that some of the missing are due to the server they reside on are decommissioned, find. But of the others I can not figure out if the FW is broken / needs updating to function. Please advise. Thank u as always.

Tags (1)
0 Karma

isoutamo
SplunkTrust
SplunkTrust

I’m afraid that there is not an easy solution for this. Probably best way is that you have boarding/integrating process where you are collecting the needed information from source systems. Or if you have any CMDB then you could utilize it to ask from owners of those systems what is the situation if you haven’t any other automated way to check those.

0 Karma

pbarbuto
Path Finder

This is actually something I'm currently working on, and you're correct when you say it isnt easy. I had to create a custom alert action which when my missing forwarder alert is triggered, will execute a python script to first ping the host to see if its available, and then also checks the hosts operation status from our CMDB. 

0 Karma

SamHTexas
Builder

Thank u sir very much & good to hear from you. I missed learning from you for a short while. Stay safe.

Tags (1)
0 Karma

pbarbuto
Path Finder

We use a lookup of all hosts reporting to splunk, and do a join with metadata for current hosts and most recent event. It allows us to see the hosts that havent reported in a day. 

| inputlookup your_host_lookup
| eval host = lower(host) 
| join host type=outer 
    [| metadata type=hosts index=_internal 
    | eval host = lower(host) 
    | sort - lastTime 
    | dedup host 
    | eval minutesSinceLastEvent = ( ( now() - recentTime ) / 60 ) 
    | eval daysSinceLastEvent = ( minutesSinceLastEvent / 1440 )] 
| where daysSinceLastEvent >= 1 
| eval lastEventDays=round(daysSinceLastEvent,2) 
| eval lastEventMins=round(minutesSinceLastEvent,0)

 

0 Karma

SamHTexas
Builder

Thank u for your response. So I need to have a complete list of all hosts in a .CSV file right? Where in Splunk do I place this file for the search to find it please? Thank u again

Tags (1)
0 Karma

pbarbuto
Path Finder

We use the Splunk kvstore. You can just have a search that runs daily to outputlookup your hosts list though. Something like this...

index=_internal | stats count by host | fields - count | outputlookup your_hosts.csv

Get Updates on the Splunk Community!

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...