Alerting

Can you help me use an alert with a Splunk search query?

kira_l
Explorer

Hi everyone.

Does anyone have any idea on how to use conditional statements within a search query?

My problem statement wants me to create an alert for as soon as the number of events in the past hour becomes less than (20% less) of the average number of events in the past 20 hours.

As of now, I have a query to parse my log data which displays the number of events.

PS : I'm pretty new to Splunk and still learning the basics. It would be great if anyone could help me with this issue.

Thanks!

0 Karma
1 Solution

jvishwak
Path Finder

You can write a query something like this:
index=_internal earliest = "-20h@h" latest = "-1h@h" | timechart count span=1h | stats avg(count) as OverAllAvgCount latest(count) as latestC | eval calcPer = ((OverAllAvgCount - latestC )/OverAllAvgCount) * 100 | where calcPer > 20

This query is comparing the last hour event count with the last 20 hours average event count, and will give result when difference is more than 20%. Then you can set alert on the events return.

View solution in original post

mbagali_splunk
Splunk Employee
Splunk Employee

Kira,

Sample search for your requirement;

index="_internal" earliest=-20h |stats count by host| eval hourlyevents=count/20 | eval avg=hourlyevents*0.2 | where hourlyevents < avg

Description:
eval hourlyevents=count/20 : (calculated no. of events per hour)
eval avg=hourlyevents*0.2 : (20% of hourly events)
where hourlyevents < avg : (If hourly events is less than 20% then alert should trigger)

You can also refer below document:

https://docs.splunk.com/Documentation/Splunk/7.2.3/SearchReference/ConditionalFunctions

kira_l
Explorer

Thanks mbagali. Appreciate it 🙂

0 Karma

jvishwak
Path Finder

You can write a query something like this:
index=_internal earliest = "-20h@h" latest = "-1h@h" | timechart count span=1h | stats avg(count) as OverAllAvgCount latest(count) as latestC | eval calcPer = ((OverAllAvgCount - latestC )/OverAllAvgCount) * 100 | where calcPer > 20

This query is comparing the last hour event count with the last 20 hours average event count, and will give result when difference is more than 20%. Then you can set alert on the events return.

kira_l
Explorer

Thanks jvishwak. This was helpful 🙂

0 Karma

kira_l
Explorer

One quick question,though. In the query , there is a statement -> latest(count) as latestC. There's another statement which gives us the time range --> _internal earliest = "-20h@h" latest = "-1h@h" . Both the "latest" are same here,right? If yes ,then does it mean that the value of latestC is the count of events of the last hour?

0 Karma

jvishwak
Path Finder

Yes, the latestC will be last hour count.

0 Karma

kira_l
Explorer

Alrighty. Thanks so much 🙂

0 Karma

mbagali_splunk
Splunk Employee
Splunk Employee

Hi Kira_l,

You can start from here : https://www.youtube.com/watch?v=SuARLqm7_jc

Steps
Navigate to the Search page in the Search and Reporting app.
Create a search.
Select Save As>Alert.
Enter a title and optional description.
Specify permissions.
Configure alert scheduling. There are two options for scheduling.
Configure trigger conditions.
(Optional) Configure a trigger throttling period.
Select one or more alert actions that should happen when the alert triggers.
Click Save.

shahbhavya
Explorer

In your Splunk Web.
1) Go to search dashboard and execute your search.
2) Then select Save As option located on the upper side of search button.
3) Select Alert
You can give title, description and alert permission as per your requirements.
In the Alert Type,
-- Scheduled will run the particular search each time when it meets the specified criteria.
-- Real-time will keep searching your query in background
After that you can specify your alert trigger condition in trigger conditions and attach actions describing what you want to do when trigger is occured.
=> in your case you can select Number of results less then some number.
You can also add custom event actions to it.
Useful link: How to creat alerts in splunk

kira_l
Explorer

Hi,
Thanks for answering this. However,my question was more about how to write the search query for the problem which I mentioned. How to use the conditional statements (like if-else in programming languages) to see if the number of events in the past hour is less than 20% of the average number of events in the past 20 hours.

0 Karma

shahbhavya
Explorer

In your Splunk Web.
1) Go to search dashboard and execute your search.
2) Then select Save As option located on the upper side of search button.
3) Select Alert
You can give title, description and alert permission as per your requirements.
In the Alert Type,
-- Scheduled will run the particular search each time when it meets the specified criteria.
-- Real-time will keep searching your query in background
After that you can specify your alert trigger condition in trigger conditions and attach actions describing what you want to do when trigger is occured.
You can also add custom event actions to it.
Useful link: How to creat alerts in splunk

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...