Splunk Enterprise logs monitoring: How do you create those alerts and assigned them to someone to be follow up on?

[tonitoagu]

I am wondering if anyone has this issue or use case. We are trying to see if we can have a system that would alert us on when a host has stopped sending logs based on the specific index it belongs. For example: We woudl like to know if a firewall has stopped sending logs within 30min and also lets say if a host for another less continuos feed has stopped, exmaple: host A of index=trickle_feed has not send in 4 hours, etc.

We are good with the logic on those searches, what i am really looking for is direction on how you create those alerts and assigned them to someone to be follow up on? what other tools you might be using for the triaging and tracking of the alert/incident/ticket/work while the feed for the Quiet host is being restored? 

[PickleRick]

There are already apps for that.

For example - https://splunkbase.splunk.com/app/4621

[tonitoagu]

I will definitely take a look at this!

Thanks so much!

[isoutamo]

With this TrackMe app you could “outsource” configuration of alerts to responsible groups from splunk admins 😉

[tonitoagu]

@gcusello thank you so much. I think i got the searches working for alerting what devices have stopped. 

My problem at this time is figuring out a way to act on those alerts. so on your example, do you konw how you would assign those hosts in the .csv to be looked and track while they are resolved? 

[gcusello]

Hi @tonitoagu,

usually in this kind of projects, it's a best practice to define a monitoring perimeter before starting the project.

But anyway, you can run a search on e.g. the last 30 days and extract all the host of your network and then store this list in the lookup.

Ciao.

Giuseppe

[tonitoagu]

Grazie Giusepppe.

[gcusello]

Hi @tonitoagu ,

if one answer solves your need, please accept one answer for the other people of Community or tell us how we can help you.

Ciao and happy splunking

Giuseppe

P.S.: Karma Points are appreciated by all the Contributors;-)

[gcusello]

HI @tonitoagu,

let me understand:you want to be alerted when events from one host stop to arrive, is this correct?

if this is your need, you an find many examples in the community (also by me!).

Anyway, you have to list all the hosts to monitor in a lookup (called e.g. perimeter.csv) containing at least one column (host) and then run a search like the following:

| metasearch index=*
| eval host=lower(host)
| stats count BY host
| append [ | inputlookup perimeter.csv | eval host=lower(host), count=0 | fields host count ]
| stats sum(count) AS total BY host
| where total=0

Ciao.

Giuseppe