Alert when firewall stops reporting to syslog

[te25]

Hello. 
I am working on creating an alert in Splunk for detecting when a firewall stops sending logs. We have all logs from firewalls forwarded to syslog in Splunk as sourcetype=pan:traffic . The problem is we have ha-pairs/ active and passive firewall and I don't see how to construct the query to check when BOTH firewalls (let's say active city-fw01 and passive city-fw02) don't send logs. We have more than 100 devices so I am using a lookup table with the list. 
Any idea would be great, thanks.

[isoutamo]

Hi

here is list of different articles how this can do in splunk.

There are a lot of options for finding hosts or sources that stop submitting events:

• Meta Woot! https://splunkbase.splunk.com/app/2949/
• TrackMe https://splunkbase.splunk.com/app/4621/
• Broken Hosts App for Splunk https://splunkbase.splunk.com/app/3247/
• Alerts for Splunk Admins ("ForwarderLevel" alerts) https://splunkbase.splunk.com/app/3796/
• Monitoring Console https://docs.splunk.com/Documentation/Splunk/latest/DMC/Configureforwardermonitoring
• Deployment Server https://docs.splunk.com/Documentation/DepMon/latest/DeployDepMon/Troubleshootyourdeployment#Forwarde...

Some helpful posts:

• https://lantern.splunk.com/hc/en-us/articles/360048503294-Hosts-logging-data-in-a-certain-timeframe
• https://www.duanewaddle.com/proving-a-negative/

[livehybrid · Accepted solution]

Hi @te25 

This might work, it assumes the lookup has a list of firewalls with a "host" field:

| inputlookup firewall_lookup.csv 
| eval reported=0 
| append 
    [ search index=yourIndex sourcetype=pan:traffic earliest=-15m 
    | stats count by host 
    | eval reported=1 
    | fields firewall reported ] 
| stats max(reported) as reported by host 
| rex field=host "(?<pair>[^0-9]+)"
| stats sum(reported) as reporting_count, values(host) as firewalls by pair 
| where reporting_count=0 
| table pair firewalls

🌟 Did this answer help you? If so, please consider:

• Adding karma to show it was useful
• Marking it as the solution if it resolved your issue
• Commenting if you need any clarification

Your feedback encourages the volunteers in this community to continue contributing

[ITWhisperer]

Have your lookup return the common name for the ha pair and detect when the pair has not sent logs (recently)