topic Re: Splunk Enterprise logs monitoring in Getting Data In

Splunk Enterprise logs monitoring: How do you create those alerts and assigned them to someone to be follow up on?

tonitoagu — Thu, 26 Jan 2023 19:32:56 GMT

I am wondering if anyone has this issue or use case. We are trying to see if we can have a system that would alert us on when a host has stopped sending logs based on the specific index it belongs. For example: We woudl like to know if a firewall has stopped sending logs within 30min and also lets say if a host for another less continuos feed has stopped, exmaple: host A of index=trickle_feed has not send in 4 hours, etc.

We are good with the logic on those searches, what i am really looking for is direction on how you create those alerts and assigned them to someone to be follow up on? what other tools you might be using for the triaging and tracking of the alert/incident/ticket/work while the feed for the Quiet host is being restored?

Re: Splunk Enterprise logs monitoring

gcusello — Thu, 26 Jan 2023 18:18:49 GMT

HI @tonitoagu,

let me understand:you want to be alerted when events from one host stop to arrive, is this correct?

if this is your need, you an find many examples in the community (also by me!).

Anyway, you have to list all the hosts to monitor in a lookup (called e.g. perimeter.csv) containing at least one column (host) and then run a search like the following:

Ciao.

Giuseppe

Re: Splunk Enterprise logs monitoring

tonitoagu — Thu, 26 Jan 2023 18:41:32 GMT

@gcusello thank you so much. I think i got the searches working for alerting what devices have stopped.

My problem at this time is figuring out a way to act on those alerts. so on your example, do you konw how you would assign those hosts in the .csv to be looked and track while they are resolved?

Re: Splunk Enterprise logs monitoring

gcusello — Thu, 26 Jan 2023 18:44:10 GMT

Hi @tonitoagu,

usually in this kind of projects, it's a best practice to define a monitoring perimeter before starting the project.

But anyway, you can run a search on e.g. the last 30 days and extract all the host of your network and then store this list in the lookup.

Ciao.

Giuseppe

Re: Splunk Enterprise logs monitoring

PickleRick — Thu, 26 Jan 2023 19:07:06 GMT

There are already apps for that.

For example - https://splunkbase.splunk.com/app/4621

Re: Splunk Enterprise logs monitoring

tonitoagu — Fri, 27 Jan 2023 12:55:12 GMT

Grazie Giusepppe.

Re: Splunk Enterprise logs monitoring

gcusello — Fri, 27 Jan 2023 13:00:43 GMT

Hi @tonitoagu ,

if one answer solves your need, please accept one answer for the other people of Community or tell us how we can help you.

Ciao and happy splunking

Giuseppe

P.S.: Karma Points are appreciated by all the Contributors;-)

Re: Splunk Enterprise logs monitoring

tonitoagu — Thu, 02 Feb 2023 14:03:14 GMT

I will definitely take a look at this!

Thanks so much!

Re: Splunk Enterprise logs monitoring

isoutamo — Thu, 02 Feb 2023 16:33:05 GMT

With this TrackMe app you could “outsource” configuration of alerts to responsible groups from splunk admins 😉