Splunk Alert when search result drop below defined threshold possibile?

[vessev]

Hello fellow splunkers,
i want to create an alert for the following search.
The search creates a statistics matrix which list the number of events from a host for the timespan defined in the search.

index=wineventlog source="WinEventLog:Security" host=testsrv1 OR host=dc* | timechart span=6h count by host limit=100

I want to define a threshold value for events in that timespan. If one of the host would drop below this threshold in my 6h timespan an alert should be triggered. There i could define a Email/SMS Messaging etc.
I've attached a picture - my goal would be to detect an unnormal behaviour like a drop or a very high peak. I'm not sure if i can have a dynamic threshold or somehting like that - but a static threshold would be good for the moment.

BR vess

[to4kawa]

| tstats count where index=wineventlog source="WinEventLog:Security" host=testsrv1 OR host=dc* by _time span=6h host
| @skeolpin method.

tstats is better for this.

[vessev]

It was the combination of @skoelpin and your answer which was a success.

Thanks!

[skoelpin · Accepted solution]

Lots and lots of ways to go about this. It all depends on skill level and complexity for what solution to implement.

If you wanted to use a static threshold, then its as simple as adding this

| eval threshold_high=150000
| eval threshold_low=1000
| eval ALERT=if(FIELD>threshold_high OR FIELD<threshold_low, "ALERT" , "GOOD")

If you wanted a more complex moving average threshold, then you can use timewrap (assuming you remove the split by host). I see you're using timechart here, this is going to add a lot of columns and also increase resource usage to monitor all these entities rather than the aggregate.

https://docs.splunk.com/Documentation/SplunkCloud/latest/SearchReference/Trendline

Option 3 is to do something more sophisticated like this which allows you to monitor each entity and have a good dynamic threshold

https://answers.splunk.com/answers/590464/how-you-detect-an-anomaly-from-a-time-frame-the-pr.html

[vessev]

I've merged the two answers.
But i'm not quite sure how to configure an "alert" when a specific filed has the string "alert" in it.

| tstats count where index=wineventlog source="WinEventLog:Security" host=s*wdc* OR host=dc-* by _time span=60min host  
| eval threshold_high=400000
| eval threshold_low=10
| eval ALERT=if( count>threshold_high OR count<threshold_low, "ALERT" , "GOOD")

[skoelpin]

You can create a custom alert condition so if it sees the field ALERT have a value of ALERT then fire. You can make it numerical instead, you can even have the alert fire if the number of results is greater than 0 etc.. Tons and tons of ways to make this work

[vessev]

I used:

search ALERT="ALERT"

as "custom" trigger, that worked pretty well for me.
I added | sort + ALERT to my search string which allowed me to see the important parts in the mail i've triggered.

Thanks for your help!

BR vess