Alerting

How to alert if a Cisco device has not sent any logs via syslog to Splunk in X hours?

bworrellZP
Communicator

Hello,

Currently I have 50 Cisco devices sending logs via syslog to Splunk and use the Cisco App. Trying to figure out how to create one alert that can fire once or twice a day, that will report all Cisco Hosts, that have not logged during that time.

Could not find an example to start with.

Thanks
Brian

0 Karma

scelikok
SplunkTrust
SplunkTrust

Hi, you can just use Broken Hosts App for Splunk

"The Broken Hosts App for Splunk is a useful tool for monitoring data going into Splunk. It has the ability to alert when hosts stop sending data into Splunk, as well as inspect the last time the final combination of data was received by Splunk."

Thanks to Hurricane Labs

If this reply helps you an upvote and "Accept as Solution" is appreciated.
0 Karma

muebel
SplunkTrust
SplunkTrust

Hi bworrellZP,

You can accomplish this by creating a search that utilizes a lookup table to define the list of expected hosts, and then search recent data to determine which hosts have actually been reporting in.

First, create a lookup like:

host,count
device1,0
device2,0
...
device50,0

Call it something like cisco_switches_list. Then create the saved search that will actually alert (assuming cisco:ios sourcetype will catch all switches:

| tstats count WHERE sourcetype=cisco:ios BY host | inputlookup append=true cisco_switches_list | stats sum(count) as count by host | search count=0

Obviously you'll have to modify the initial tstats search to be limited to the expected hosts, but then you can see that the lookup is appended, if any hosts didn't show up in the initial search they end up have a count of 0, which then gets caught at the final search. You then create an alert for whenever this search returns any results, indicating a missing host

Please let me know if this answers your question!

bworrellZP
Communicator

Not sure I understand the count column in the lookup table. Can you help me understand that, then I will give it a shot.

Thanks

0 Karma

muebel
SplunkTrust
SplunkTrust

sure, it serves as an indicator of a missing host. The main search does a count of events for the hosts, but will only do so for hosts that have events. Setting this lookup ensures that if the host is missing from the results, it will be there with a count of 0, which indicates that it hasn't been sending anything in.

0 Karma

bworrellZP
Communicator

so I created a csv, called cisco_lookup, then I added it to the lookups in Splunk. Also added it to the lookup defs, but left it with defaults, other Tried the search:

| tstats count WHERE sourcetype=cisco:asa BY host | inputlookup append=true cisco_lookup | stats sum(count) as count by host | search count=0

What I get when I search is everything, and they all have a zero. Am I missing a step or did I need to configure the advanced options in the look up defs?

0 Karma

Jeremiah
Motivator

How about using the metadata command?

| metadata type=hosts index=cisco | where recentTime<(now()-(3600)) | convert ctime(*Time)

You can also restrict the host pattern if you need to (for example if your devices are intermingled in the same index).

| metadata type=hosts  | where recentTime<(now()-(3600))  AND like(host,"%cisco%") | convert ctime(*Time)
0 Karma

bworrellZP
Communicator

They are all intermingled in an index (cisco-network). That could be the issue why I could not find an example.

But with 5 or 6 devices per site, thought an email that 172.16.X.X, 192.168.X.X and 10.X.X.X have not logged in the last 6 hours, would be best

0 Karma

Jeremiah
Motivator

How do you identify them as cisco devices?

0 Karma

bworrellZP
Communicator

Sorry I should have been more clear, the index is ONLY Cisco devices, but has many types of Cisco devices all mixed together (Routers, Switches, and firewalls).

0 Karma

Jeremiah
Motivator

So of those, you have 50 particular devices you want to be notified about? If that's the case, then you'll need to have some way of identifying those devices. As muebel suggested, you probably need a lookup.

0 Karma

bworrellZP
Communicator

Each device does show up as a unique host. My thought was to use something like this (granted not sure this is correct) sourcetype="cisco:asa" | stats latest(_time) as _time by host| reltime but how to alert on that.

Using a lookup is possible, as I noted below, not sure I understand the Count column in his example.

0 Karma
Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

Splunk is officially part of Cisco

Revolutionizing how our customers build resilience across their entire digital footprint.   Splunk ...

Splunk APM & RUM | Planned Maintenance March 26 - March 28, 2024

There will be planned maintenance for Splunk APM and RUM between March 26, 2024 and March 28, 2024 as ...