Splunk Search

when using predict fields become null

kiamco
Path Finder

so when I use the predict command my fields become null

index=summary source="summary_events_2"
orig_source=pnr
ms_level=ERROR OR ms_level=error
NOT event=no-event
| timechart span=5m sum(count) as count
| predict count as prediction algorithm=LLP future_timespan=200 holdback=0
| eval residual=count-round(prediction,0)
| streamstats window=200 current=true median(residual) as median_of_residual
| eval abs_dev=(abs(residual - median_of_residual))
| streamstats window=200 current=true median(abs_dev) as median_abs_dev
| eval upper_bound=(median_of_residual + median_abs_dev * 20)
| eval anomaly=if(residual > upper_bound,1,0)

so this is my query and I want to add

|table event, anomaly, count

but for some reason the "event" field is null. can anyone explain why?

0 Karma
1 Solution

Sukisen1981
Champion

Hi,

You are applying a table command after a timechart and prediction command. There is actually, no event command from your timechart - your timechart (and the subsequent fields like residual,anomaly etc.) are all based on evals of count. There is no way events are going to be captured AFTER you apply a timechart command and just use the count , based on the timechart for further processing.
For example, if i write something like this | eval event="vvvv" | table |table event, anomaly, count , then you will receive an event field with vvv as the field values, see what I mean?
It is difficult to say what actually you want under the event field , but if I just slightly modify your timechart command to something like this - |timechart span=5m sum(count) as count, values(_raw) as event and then run the rest of your query (including |table event,anomaly,count) you WILL get values under the event field (event =_raw) in this case.
My example is bad and the query takes a long time to execute, try running this for last 15 mins. What i did was select the _audit index so that you can use the code as it is, run this code for the lats 15 minutes and see the otuput index="_audit" | timechart span=5m sum(count) as count, values(_raw) as event
| predict count as prediction algorithm=LLP5 future_timespan=200 holdback=0
| eval residual=count-round(prediction,0)
| streamstats window=200 current=true median(residual) as median_of_residual
| eval abs_dev=(abs(residual - median_of_residual))
| streamstats window=200 current=true median(abs_dev) as median_abs_dev
| eval upper_bound=(median_of_residual + median_abs_dev * 20)
| eval anomaly=if(residual > upper_bound,1,0)|table event, anomaly, count

So, either you need to include what you mean by event in the timechart(your prediction for count will remain unscathed ) OR define something using an eval for the event to pick values

View solution in original post

0 Karma

Sukisen1981
Champion

Hi,

You are applying a table command after a timechart and prediction command. There is actually, no event command from your timechart - your timechart (and the subsequent fields like residual,anomaly etc.) are all based on evals of count. There is no way events are going to be captured AFTER you apply a timechart command and just use the count , based on the timechart for further processing.
For example, if i write something like this | eval event="vvvv" | table |table event, anomaly, count , then you will receive an event field with vvv as the field values, see what I mean?
It is difficult to say what actually you want under the event field , but if I just slightly modify your timechart command to something like this - |timechart span=5m sum(count) as count, values(_raw) as event and then run the rest of your query (including |table event,anomaly,count) you WILL get values under the event field (event =_raw) in this case.
My example is bad and the query takes a long time to execute, try running this for last 15 mins. What i did was select the _audit index so that you can use the code as it is, run this code for the lats 15 minutes and see the otuput index="_audit" | timechart span=5m sum(count) as count, values(_raw) as event
| predict count as prediction algorithm=LLP5 future_timespan=200 holdback=0
| eval residual=count-round(prediction,0)
| streamstats window=200 current=true median(residual) as median_of_residual
| eval abs_dev=(abs(residual - median_of_residual))
| streamstats window=200 current=true median(abs_dev) as median_abs_dev
| eval upper_bound=(median_of_residual + median_abs_dev * 20)
| eval anomaly=if(residual > upper_bound,1,0)|table event, anomaly, count

So, either you need to include what you mean by event in the timechart(your prediction for count will remain unscathed ) OR define something using an eval for the event to pick values

View solution in original post

0 Karma

kiamco
Path Finder

it works!! thanks a lot for the very detailed answer

0 Karma

richgalloway
SplunkTrust
SplunkTrust

When you can't figure out where a field gets "lost", drop one command at a time from the query until the field appears. The last command dropped is the culprit.

---
If this reply helps you, an upvote would be appreciated.
0 Karma

kiamco
Path Finder

yup did that and it gets lost after the predict command in the query. I'm just wondering why or is there any way to get those fields back?

0 Karma
.conf21 Now Fully Virtual!
Register for FREE Today!

We've made .conf21 totally virtual and totally FREE! Our completely online experience will run from 10/19 through 10/20 with some additional events, too!