How to alert if a Cisco device has not sent any logs via syslog to Splunk in X hours?

[bworrellZP]

Hello,

Currently I have 50 Cisco devices sending logs via syslog to Splunk and use the Cisco App. Trying to figure out how to create one alert that can fire once or twice a day, that will report all Cisco Hosts, that have not logged during that time.

Could not find an example to start with.

Thanks
Brian

[scelikok]

Hi, you can just use Broken Hosts App for Splunk

"The Broken Hosts App for Splunk is a useful tool for monitoring data going into Splunk. It has the ability to alert when hosts stop sending data into Splunk, as well as inspect the last time the final combination of data was received by Splunk."

Thanks to Hurricane Labs

[muebel]

Hi bworrellZP,

You can accomplish this by creating a search that utilizes a lookup table to define the list of expected hosts, and then search recent data to determine which hosts have actually been reporting in.

First, create a lookup like:

host,count
device1,0
device2,0
...
device50,0

Call it something like cisco_switches_list. Then create the saved search that will actually alert (assuming cisco:ios sourcetype will catch all switches:

| tstats count WHERE sourcetype=cisco:ios BY host | inputlookup append=true cisco_switches_list | stats sum(count) as count by host | search count=0

Obviously you'll have to modify the initial tstats search to be limited to the expected hosts, but then you can see that the lookup is appended, if any hosts didn't show up in the initial search they end up have a count of 0, which then gets caught at the final search. You then create an alert for whenever this search returns any results, indicating a missing host

Please let me know if this answers your question!

[bworrellZP]

Not sure I understand the count column in the lookup table. Can you help me understand that, then I will give it a shot.

Thanks

[muebel]

sure, it serves as an indicator of a missing host. The main search does a count of events for the hosts, but will only do so for hosts that have events. Setting this lookup ensures that if the host is missing from the results, it will be there with a count of 0, which indicates that it hasn't been sending anything in.

[bworrellZP]

so I created a csv, called cisco_lookup, then I added it to the lookups in Splunk. Also added it to the lookup defs, but left it with defaults, other Tried the search:

| tstats count WHERE sourcetype=cisco:asa BY host | inputlookup append=true cisco_lookup | stats sum(count) as count by host | search count=0

What I get when I search is everything, and they all have a zero. Am I missing a step or did I need to configure the advanced options in the look up defs?

[Jeremiah]

How about using the metadata command?

| metadata type=hosts index=cisco | where recentTime<(now()-(3600)) | convert ctime(*Time)

You can also restrict the host pattern if you need to (for example if your devices are intermingled in the same index).

| metadata type=hosts  | where recentTime<(now()-(3600))  AND like(host,"%cisco%") | convert ctime(*Time)

[bworrellZP]

They are all intermingled in an index (cisco-network). That could be the issue why I could not find an example.

But with 5 or 6 devices per site, thought an email that 172.16.X.X, 192.168.X.X and 10.X.X.X have not logged in the last 6 hours, would be best

[Jeremiah]

How do you identify them as cisco devices?

[bworrellZP]

Sorry I should have been more clear, the index is ONLY Cisco devices, but has many types of Cisco devices all mixed together (Routers, Switches, and firewalls).

[Jeremiah]

So of those, you have 50 particular devices you want to be notified about? If that's the case, then you'll need to have some way of identifying those devices. As muebel suggested, you probably need a lookup.

[bworrellZP]

Each device does show up as a unique host. My thought was to use something like this (granted not sure this is correct) sourcetype="cisco:asa" | stats latest(_time) as _time by host| reltime but how to alert on that.

Using a lookup is possible, as I noted below, not sure I understand the Count column in his example.