All Apps and Add-ons

Splunk Alert

gajananh999
Contributor

Dear All,

We are using splunk for monitoring purpose. We are pulling data from multiple application We want fire an alert suppose if i wont get the one statement in particular log file. I get a one event at every two hour so whenever i get that event i get a alert. But my requirement is different if i wont get this event in 2 hours then i have to get a alert . Can anybody help me how to get this done?

Thanks

Gajanan Hiroji

0 Karma
1 Solution

martin_mueller
SplunkTrust
SplunkTrust

You could have your alert trigger whenever there are zero results. For example, run this realtime with a two-hour window:

your search terms | stats count

Configure the alert's trigger condition to fire when count is zero. Make sure to also set throttling.

View solution in original post

renems
Communicator

Also, you could evaluate the search, by calculating a "gap". Then alert when your gap is too long.
This is how I did it:

| stats max(_time) As LatestTime by appserver | eval gap=(now()-LatestTime) | eval appserver="WebSphere" | sort num(gap) D | head 1 | rangemap field=gap low=0-300 default=severe

renems
Communicator

But Martin is right, since you want to alert after 2 hours, your search has to span at least 2 hours. I don't know what amount of data you have, but that could be an expensive search if it would span a lot of buckets.

0 Karma

renems
Communicator

with the "| eval gap=(now()-LatestTime)" you create a field that sets a value for the time that has elapsed since your event was recorded. In this example, the rangemap will turn to red if "gap" is greater then 300 secs.

0 Karma

gajananh999
Contributor

Thanks Renems. Can you elaborate this i am not able to get the | eval appserver="WebSphere" | sort num(gap) D | head 1 | rangemap field=gap low=0-300 default=severe

0 Karma

martin_mueller
SplunkTrust
SplunkTrust

Note, you need to use a time range longer than two hours for this to work... Long enough to be sure that the most recent event is still visible but older than two hours when running the alert.

martin_mueller
SplunkTrust
SplunkTrust

You could have your alert trigger whenever there are zero results. For example, run this realtime with a two-hour window:

your search terms | stats count

Configure the alert's trigger condition to fire when count is zero. Make sure to also set throttling.

gajananh999
Contributor

Yes if the event doesn't come in 2 hour then i have fire a alert.

0 Karma

martin_mueller
SplunkTrust
SplunkTrust

Ah. Select cron schedule and enter this:

0 */2 * * *

That'll run at 00:00, 02:00, 04:00, etc.

If there's an event at 00:30 and one at 03:00 that's more than two hours... should your alert fire in this case? Scheduling this with a two-hour time range every two hours would not fire because there was an event in each two-hour time range despite the difference between the events being greater than two hours.

0 Karma

gajananh999
Contributor

Whenever i am scheduling alert the option for time range is 1) Every hour 2) every day kind of thing but there is no every two hour option where i have to do this conf files?

0 Karma

martin_mueller
SplunkTrust
SplunkTrust

I don't think I understand that question, please elaborate on what you're asking.

0 Karma

gajananh999
Contributor

Thanks for your reply but when there is alert we can schedule for 2 hours in splunk.?

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...