topic Re: Splunk Alert when no data is being send. in Other Usage

Splunk Alert when no data is being send.

kenncaid — Sat, 15 Jul 2023 09:36:53 GMT

I have an IP which is sending sending and receiving traffic displayed in timechart:

192.168.1.1 | timechart c by avg(bytes)

if this IP is stop sending traffic , how do i setup an alert for that?

i searched in many topics but cannot find the solution.

Re: Splunk Alert when no data is being send.

gcusello — Sat, 15 Jul 2023 09:45:55 GMT

Hi @kenncaid,

at first you should identify the minimum frequency that you receive logs from that endpoint, to avoid to receive false positives and the field containing the ip.

then if the minimum frequency is 30 minutes and the field is called "IP", you should run a simple search like the following:

index=your_index IP="192.168.1.1" earliest=-35m@m latest=-5m

then, you should save this search (using the "Save AS" button in the top right of the search dashboard, then adding the requested information:

condition: 0 results,
scheduling using cron with "*/30 * * * *"

Ciao.

Giuseppe

Re: Splunk Alert when no data is being send.

kenncaid — Sat, 15 Jul 2023 10:00:49 GMT

You mean saving as an Alert ? which settings need to be changed in order for me to setup the email notification?

the logs are realtime

Re: Splunk Alert when no data is being send.

PickleRick — Sat, 15 Jul 2023 12:53:04 GMT

1) You have alert actions at the bottom - you can add an email notification action there

2) Don't use real-time alerts. Real-time searches are generally bad for your environment - they hog a whole CPU for themselves reducing your environment's capacity. There are very few cases where real-time searches make sense.

Re: Splunk Alert when no data is being send.

kenncaid — Sat, 15 Jul 2023 13:42:13 GMT

Thanks. I was aware of the email setup.

Can you provide optimal settings for the non realtime settings? I want an email as soon as the ip stops sending traffic. (It reaches zero events)

Re: Splunk Alert when no data is being send.

PickleRick — Mon, 17 Jul 2023 07:27:44 GMT

Heh. That's the trick - everyone wants "as soon as..." but if you ask "what do you mean by that", it turns out it ain't that easy.

Because "zero events" must be counted across some time range. Otherwise it would not make sense at all - between every two events received there is a period with zero events. So that's one thing - you always have some sampling interval.

Anyway, if you really really really have a need for a near-realtime downtime detection, you probably need to turn to other tools which will help you achieve that. Unfortunately, all monitoring comes with a price. Typically in a form of additional system load and overall monitoring overhead. There's a great book from O'Reilly on that topic - https://www.oreilly.com/library/view/effective-monitoring-and/9781449333515/

Anyway, the typical way to go is to run a scheduled search (with frequency depending on your system load and alerting needs).

Re: Splunk Alert when no data is being send.

gcusello — Mon, 17 Jul 2023 09:21:02 GMT

Hi @kenncaid,

yes, you have only (as @PickleRick said) to add an action to your alert (e.g. an eMail).

In addition I agree with @PickleRick in avoiding real time searches, correlate the executing frequency with the timeframe in your search.

Ciao.

giuseppe