Predict: show past events and future predictions

[nplamondon]

I'm using predict, and seeing good results, but I would like to clean up my visualization.

What I would like is to see past data and future predicted data with no overlap.

Using eval predicted=if(isnull(foo), predicted, null()), I am able to show predicted data only where actual data points don't exist, but I still see the upper95 and lower95 overlaid on top of actual data. Trying an equivalent eval for "lower95(predicted)" doesn't work.

Is there a way I can show only actual data where it exists, and only predicted data (with probability ranges) where it doesn't?

[nplamondon · Accepted solution]

Good news, everybody!

This was entirely a PEBKAC issue. Working on this more today, I found the "Null values" option under formatting. No idea how I had missed that previously. Setting "Null values" to "connect" made the leading zeroed line disappear.

Thanks, again, for your help, Rich and Clara!

[rflouquet]

Hi @nplamondon !

Sorry but I don't understand how you fired it out... It realy would help me !

Thanks

[nplamondon]

• Open your dashboard, click Edit.
• On the panel, click the paint brush
• Change Null Values to "Connect"
• Close the edit dialog
• Click Save at the top of the dashboard

[Richfez]

You are welcome for what tiny glimmers of help I gave, oh "Everyone's Nate".

[Richfez]

Run anywhere:

* 
| timechart count as foo 
| predict foo AS predicted_foo 
| eval upper95(predicted_foo)=if(_time<=now(), predicted_foo, 'upper95(predicted_foo)' )
| eval lower95(predicted_foo)=if(_time<=now(), predicted_foo, 'lower95(predicted_foo)' )

Of course change the time frame to something reasonable (last hour, last day, whatever). NOTE that the upper95(predicted_foo) in the eval has to be inside single quotes, otherwise it'll think it's a function or something, so be careful about that syntax.

That gives, on my data, something like

If you have any problems applying that to your own data, let us know because I'm sure we can help!

Happy Splunking!

[nplamondon]

Thanks, Rich. That gets me much closer.

The only issue remaining is that I can't seem to get rid of the lower95. Your solution eliminates the upper95 just fine, but lower95 goes to 0. I'm wondering if I stumbled onto a bug...

| eval predicted=if(isnull(credits), round(predicted, 0), null())
| eval upper95(predicted)=if(_time<=now(), null(), 'upper95(predicted)' )
| eval lower95(predicted)=if(_time<=now(), null(), 'lower95(predicted)' )

[Richfez]

That's interesting.

Surely there's a typo? A special character embedded in there or something?

Try
1) Changing the order of the two items around.
2) Get rid of the lower95 part, copy and paste the upper95 again and manually change the "upper" to "lower" in each spot needed.

Number one could be interesting - will lower work and upper not now? Inquiring minds want to know!
Number two should really settle things though. I'm SURE there's just something goofy going on with some space or something.

[nplamondon]

I just noticed that my screencaps haven't been posting. Trying again...

I also tried adding | eval lower95(predicted)=if('lower95(predicted)'=0, null(), 'lower95(predicted)') for s&g's, but it was still set to 0.

Looking at the Statistics tab, there are no values where I expect lower95 to be null, which is correct, implying this is a display bug.

[Richfez]

Agreed it seems like a UI bug given that last information. I can try to recreate this again, make sure it wasn't a fluke it worked for my screenshot a few days ago.

What version of Splunk is the SH you are using?

My screenshot is from a 6.5.2 on Windows (on my laptop as a test box) using Firefox.

Let me recreate something here on 6.5.2 on Linux using various browser, see if I can make it break or not.

[Richfez]

Tested and confirmed I do NOT have this problem.

index=network earliest=-5m latest=-15s 
| bin _time span=15s 
| stats count AS foo by _time 
| predict foo AS predicted_foo
| eval upper95(predicted_foo)=if(_time<=now(), predicted_foo, 'upper95(predicted_foo)' )
| eval lower95(predicted_foo)=if(_time<=now(), predicted_foo, 'lower95(predicted_foo)' )

This has predicted_foo all the way across the Line Chart (as expected, I do nothing with it) with no upper/lower bounds where foo exists. Around where foo stops the upper/lower bounds of predicted_foo appear.

That's Firefox (latestish) on Windows running against a 6.5.2 Search Head.
It also works on IE in the same way.
And on Chrome.

Weird. Well anyway, I think there's a bug somewhere. Just not sure where it is because I can't find it. 🙂

When you find a proper answer (bug, fixed in 6.5.3, whatever) update us and mark something as Accepted!

[nplamondon]

Just for completeness (and my sanity), would you please run your query setting upper/lower to null(), rather than predicted_foo? It seems to me the problem appears when nulling lower95.

[Richfez]

Sorry, I hadn't noticed this reply.

So, interesting. There is a UI bug in here, or at least a odd behavior. There are several permutations of this:

Using null() for neither and sticking with just making each match our prediction, using null() on just upper95, using null() for lower95, or using null() for both.

Using null() for neither "correction" is our default, works fine.

Using null() for lower is the interesting one, seems that no matter what I do it shades from zero to the value for all locations where the value is null. You'd think that the "format" option would be able to fix this (on how to handle nulls), but it can't. THAT I believe is certainly a bug.

Using null() for upper is not a problem. A slightly different behavior at the point where they become non-predict_foo, because they're not following the predict_foo up until then they don't "Expand out" from predict_foo but instead just "appear".

Using null() for BOTH is also not a problem - Same behavior as above, but they both work.

It's only when ONLY lower95 is replaced with null. So, for sure a bug of some sort.

[Richfez]

After that, I saw your answer. LOL.

I even just went through and came up with screenshots of all 4 - I get odd behavior of the lower95 replacement (when used alone) regardless of settings of the formatting of null in the visualization.

I may open a ticket and see what happens with mine, then.

[nplamondon]

This is Splunk 6.5.2 running on RHEL 6.8.

Since this reeks of display bug, I'll submit one. Thanks for helping me poke at this!

[nplamondon]

Good ideas.

1) Swapping the upper/lower lines did not change the result.
2) I had already done this, thinking I had made a typo. Just for completeness, I tried it again, just now. No change.

[cmerriman]

can you just do a |fields - lower95 - upper96 or am I misunderstanding something else?

[nplamondon]

That would work for eliminating those fields totally, but I'd like to see them for the predicted portion.

[cmerriman]

what if you did the same thing for lower and upper that you do for predicted?

|eval upper95=if(isnull(foo), upper95, null())|eval lower95=if(isnull(foo), lower95, null())

[nplamondon]

`index=xxxxx sourcetype=duo:info_summary earliest=-7d

| timechart span=1h min(telephony_credits_remaining) as credits

| predict future_timespan=168 credits as predicted

| eval predicted=if(isnull(credits), round(predicted, 0), null())`

The fields I'm concerned with are named "upper95(predicted)" and "lower95(predicted)". If I try

| eval predicted=if(isnull(credits), round(predicted, 0), null()), "lower95(predicted)"=if(isnull(credits), "lower95(predicted)", null()), "upper95(predicted)"=if(isnull(credits), "upper95(predicted)", null())

those fields are 0 through the entire graph. If I don't include the quotes, I get an error calling the upper95 function (which isn't a function).

[somesoni2]

We need to see you full query to understand what all fields are available. It would be easier then to correlate to your description.