Splunk Search

Predict Command - Alert when value breaches upper95

MikeElliott
Communicator

Hi All,

I'm trying to write a search that looks at creating an alert where there is a significant spike in HTTP POST requests.

I am interested in using the predict command and alerting where the total count(http_request) (where http_request=POST) requests by source_ip breaches the predicted upper95.

In theory, it would look something like:

index=web_proxy 
| search http_request=POST 
| stats count(http_request) AS POST_Count by source_ip 
| predict POST_Count by source_ip 
| where POST_Count >= upper95

Any assistance, or pointers, would be greatly appreciated.

1 Solution

MikeElliott
Communicator

Apologies for the delay in responding. I was able to resolve the issue with the below logic. The index has been swapped out for a generic term, ofc 😉

index="webproxy" requestmethod=POST earliest=-65m@m latest=-5m@m 
| timechart span=1m count as POST_Requests 
| predict POST_Requests as Predicted_Requests 
| rename upper95(Predicted_Results) as Ceiling 

View solution in original post

0 Karma

MikeElliott
Communicator

Apologies for the delay in responding. I was able to resolve the issue with the below logic. The index has been swapped out for a generic term, ofc 😉

index="webproxy" requestmethod=POST earliest=-65m@m latest=-5m@m 
| timechart span=1m count as POST_Requests 
| predict POST_Requests as Predicted_Requests 
| rename upper95(Predicted_Results) as Ceiling 
0 Karma

MikeElliott
Communicator

This is some really cool logic that can be adapted to detect all sorts of spikes - Recently we have deployed this for spikes in DNS traffic.

0 Karma

dauren_akilbeko
Communicator

The predict command must be preceded by the timechart command. The predict command requires time series data.

For more info: https://docs.splunk.com/Documentation/Splunk/latest/SearchReference/Predict

MikeElliott
Communicator

Damn - I was really hoping that this wouldn't be the case...

Okay, so we can use timechart I suppose - Any suggestions on how to get the timechart to display count(http_request) as POST_Count by source_ip, or am I asking a bit much?

0 Karma

dauren_akilbeko
Communicator

The problem with predict, is that you can't use wildcard. Not very efficient way to this https://answers.splunk.com/answers/661506/predict-with-wildcard.html

0 Karma

MikeElliott
Communicator

Hi dauren,

Apologies for the delay in getting back. I have posted the logic I ended up going with below - Since you were definitely instrumental in getting there, if you wanna post the logic, I'd be happy to mark as an answer 🙂

0 Karma

MikeElliott
Communicator

When attempting to run the above search, I get the error message External search command 'predict' returned error code 1.

0 Karma
Get Updates on the Splunk Community!

New in Observability - Improvements to Custom Metrics SLOs, Log Observer Connect & ...

The latest enhancements to the Splunk observability portfolio deliver improved SLO management accuracy, better ...

Improve Data Pipelines Using Splunk Data Management

  Register Now   This Tech Talk will explore the pipeline management offerings Edge Processor and Ingest ...

3-2-1 Go! How Fast Can You Debug Microservices with Observability Cloud?

Register Join this Tech Talk to learn how unique features like Service Centric Views, Tag Spotlight, and ...