topic Re: Forecast Time series in Splunk Search

Forecast Time series

BenImen — Tue, 29 Sep 2020 20:38:42 GMT

Hi Ninjas,

I have a query that looks like this:

sourcetype="x" index=y source="z" host="S" 
| bin _time span=10m
| stats dc(CN) as Actual by _time | lookup CN_Forecast_S.csv _time OUTPUT lowerBound pred upperBound 
| eval isOutlierLow=if(Actual < lowerBound , abs(Actual-lowerBound)/lowerBound, 0) 
| eval isOutlierHigh=if(Actual > upperBound, abs(Actual-upperBound)/upperBound, 0)  
| eval isOutlier=if(Actual < lowerBound OR Actual > upperBound, abs(Actual)/abs(upperBound-lowerBound), 0) 
| fields _time, Actual, lowerBound, pred, upperBound, isOutlier, isOutlierLow, isOutlierHigh

The CN_Forecast_S.csv is a lookup file generated by a savedsearch that predict +2days of data.
The problem is my query display data until "now" only and I would like to show data for the rest +xdays that I already have predicted in the same graph. I tried to specify lastest=+2d@d , but that didn't work
That's the result of my query

Thank you in advance.

Re: Forecast Time series

skoelpin — Wed, 25 Jul 2018 17:01:33 GMT

Why are you pulling boundaries from a lookup table? Why not calculate it dynamically or push it into a summary index? Your current method must be pretty slow

This wouldn't really qualify as machine learning as your not dynamically populating boundaries based on previous trends. You should let the machine do the work and calculate the boundaries based off historical values rather than storing them in a lookup table. You're going to get lots of Type I and Type II errors with this method

Re: Forecast Time series

BenImen — Thu, 26 Jul 2018 08:06:32 GMT

I'm populating boundaries dynamically based on previous trends and I'm putting them into a local file via outputlookup (because I don't have the permission to use the Collect command so I can't store the data into a summary index).
The question is how to continue to display the boundaries in the same chart from "now" to +2days. Thank you 🙂

Re: Forecast Time series

skoelpin — Thu, 26 Jul 2018 14:16:44 GMT

This is not an ideal way, you should either get access to the collect command or enable scheudled reports to populate the summary index.

If you want to push the boundaries into the future, you should use the timeshift technique

| eval w=case( (_time>relative_time(now(), "+1d@d-5w-30m") AND _time<=relative_time(now(), "+1d@d-5w+1d+30m")), 5, (_time>relative_time(now(), "+1d@d-4w-30m") AND _time<=relative_time(now(), "+1d@d-4w+1d+30m")), 4, (_time>relative_time(now(), "+1d@d-3w-30m") AND _time<=relative_time(now(), "+1d@d-3w+1d+30m")), 3, (_time>relative_time(now(), "+1d@d-2w-30m") AND _time<=relative_time(now(), "+1d@d-2w+1d+30m")), 2, (_time>relative_time(now(), "+1d@d-1w-30m") AND _time<=relative_time(now(), "+1d@d-1w+1d+30m")), 1) 
| eval shift=case(isnotnull(w),"+"+w+"w-30m,+"+w+"w-20m,+"+w+"w-10m,+"+w+"w-0m,+"+w+"w+10m,+"+w+"w+20m,+"+w+"w+30m,") 
| where isnotnull(shift) 
| makemv delim="," shift 
| mvexpand shift 
| eval time=relative_time(_time,shift)

Re: Forecast Time series

BenImen — Tue, 29 Sep 2020 20:39:32 GMT

Yes I already doing this:
sourcetype="X" index=Y source="Z" host="S"
| bin _time span=10m
| stats dc(CN) as Actual by host, _time
| Forecast4w(Actual,90.0,+1d,1) | outputlookup CN_Forecast_S.csv append=True
The Forcast4w is a macro that contain the timeshift technique, and I scheduled this to run every day

Re: Forecast Time series

skoelpin — Thu, 26 Jul 2018 15:54:51 GMT

So you should be pushing it 1 day into the future already then..

Re: Forecast Time series

BenImen — Tue, 31 Jul 2018 10:13:08 GMT

It's a display issue: my current query show the "pred" just until now and I would like that continue to display the rest of the data generated by the Forcast4w macro. I didn't get the result desired by specifying the Latest time +1d@d. Sorry if i wasn't clear enough!

Re: Forecast Time series

skoelpin — Tue, 31 Jul 2018 13:48:32 GMT

So your issue is solved then? Can you accept the answer to close out the question?

Re: Forecast Time series

BenImen — Tue, 31 Jul 2018 14:07:40 GMT

Not yet, still have this issue!

Re: Forecast Time series

skoelpin — Tue, 31 Jul 2018 14:14:45 GMT

I'm not clear on your problem. What is not working properly?