topic Re: Predict Command - Alert when value breaches upper95 in Splunk Search

Predict Command - Alert when value breaches upper95

MikeElliott — Fri, 10 Aug 2018 07:15:59 GMT

Hi All,

I'm trying to write a search that looks at creating an alert where there is a significant spike in HTTP POST requests.

I am interested in using the predict command and alerting where the total count(http_request) (where http_request=POST) requests by source_ip breaches the predicted upper95.

In theory, it would look something like:

index=web_proxy 
| search http_request=POST 
| stats count(http_request) AS POST_Count by source_ip 
| predict POST_Count by source_ip 
| where POST_Count >= upper95

Any assistance, or pointers, would be greatly appreciated.

Re: Predict Command - Alert when value breaches upper95

MikeElliott — Fri, 10 Aug 2018 07:17:03 GMT

When attempting to run the above search, I get the error message External search command 'predict' returned error code 1.

Re: Predict Command - Alert when value breaches upper95

dauren_akilbeko — Fri, 10 Aug 2018 07:25:17 GMT

The predict command must be preceded by the timechart command. The predict command requires time series data.

For more info: https://docs.splunk.com/Documentation/Splunk/latest/SearchReference/Predict

Re: Predict Command - Alert when value breaches upper95

MikeElliott — Fri, 10 Aug 2018 07:36:07 GMT

Damn - I was really hoping that this wouldn't be the case...

Okay, so we can use timechart I suppose - Any suggestions on how to get the timechart to display count(http_request) as POST_Count by source_ip, or am I asking a bit much?

Re: Predict Command - Alert when value breaches upper95

dauren_akilbeko — Fri, 10 Aug 2018 09:08:22 GMT

The problem with predict, is that you can't use wildcard. Not very efficient way to this https://answers.splunk.com/answers/661506/predict-with-wildcard.html

Re: Predict Command - Alert when value breaches upper95

MikeElliott — Wed, 14 Nov 2018 02:42:25 GMT

Apologies for the delay in responding. I was able to resolve the issue with the below logic. The index has been swapped out for a generic term, ofc 😉

index="webproxy" requestmethod=POST earliest=-65m@m latest=-5m@m 
| timechart span=1m count as POST_Requests 
| predict POST_Requests as Predicted_Requests 
| rename upper95(Predicted_Results) as Ceiling

Re: Predict Command - Alert when value breaches upper95

MikeElliott — Wed, 14 Nov 2018 02:42:58 GMT

This is some really cool logic that can be adapted to detect all sorts of spikes - Recently we have deployed this for spikes in DNS traffic.

Re: Predict Command - Alert when value breaches upper95

MikeElliott — Wed, 14 Nov 2018 02:43:57 GMT

Hi dauren,

Apologies for the delay in getting back. I have posted the logic I ended up going with below - Since you were definitely instrumental in getting there, if you wanna post the logic, I'd be happy to mark as an answer 🙂