topic Re: Splunk Alert for spike in log events in Alerting

How to create a Splunk Alert for spike in log events?

testingtena — Wed, 11 May 2022 22:46:48 GMT

Trying to implement an alert on detecting spikes in logged events in our Splunk deployment and not sure how to go about it...

For example: Have 15 hosts with varying levels of sources within each... one of my sources in a host averages about 5-6k events per day over the past 30 days; then out of the blue, we're hit with 1.3 million events on one of the days.

Assuming the alert would need to be tailored to each host (or source, not sure) and would need an average number of events over a "normal" week to compare to when there's a spike?

Any help would be greatly appreciated.

Re: Splunk Alert for spike in log events

ITWhisperer — Thu, 14 Oct 2021 16:57:27 GMT

Something like this:

Re: Splunk Alert for spike in log events

testingtena — Thu, 14 Oct 2021 17:28:04 GMT

Thanks for the quick reply @ITWhisperer but when I try running it I get the following.

Re: Splunk Alert for spike in log events

ITWhisperer — Thu, 14 Oct 2021 17:33:16 GMT

Well it was just an example - you probably want to add an index or more to restrict your search depending on your actual data - similarly, it looks like you don't have host extracted so change this for something you do have that you want to group your data by - only you will know what this is as you didn't provide that information in your original post.

Re: Splunk Alert for spike in log events

Taruchit — Wed, 11 May 2022 10:54:48 GMT

Thank you for sharing your inputs and code logic.

I tried the below way: -
In the time range picker: "earliest: -7d@d" "latest:now".

And I get results for each sourcetype when the count for a sourcetype on a given date is greater than average count by 25%.

I need your assistance to build SPL which takes average of past 15 days and compare it with today's results, but it should exclude today's date in the average. For example: - today is 11 May 2022, the past 15 days should be from 26 April 2022 to10 May 2022

Re: Splunk Alert for spike in log events

ITWhisperer — Wed, 11 May 2022 11:20:48 GMT

You could do something like this (with timepicker at -15d@d)

Re: Splunk Alert for spike in log events

Taruchit — Wed, 11 May 2022 14:03:51 GMT

Thank you for your prompt inputs.

I tried with following: -

index=xxx sourcetype IN ("A") | bin _time span=1d | stats count by _time sourcetype | eval previous=if(_time<relative_time(now(),"@d"),count,null())

Due to large volume of data, for testing purpose I only kept once sourcetype in the SPL and time range as Last 7 days.

In output, I get table with following columns: -
_time
sourcetype
count
previous

I get results for each date in past 7 days, however the values under column count and previous are same.
Sample output: -

_time	sourcetype	count	previous
2022-05-04	A	1004558705	1004558705
2022-05-05	A	2450936208	2450936208
2022-05-06	A	3074060943	3074060943

Thus, can you please help me to correct where I am going wrong.