Splunk Search

Why are streamstats reset_after for streamstats aggregation not working?

patientsplunker
Loves-to-Learn Everything

Hello,

I am trying to use Streamstats with Sum(value) and I want to reset that sum after it reaches a certain threshold (in the example below this threshold is 1000). The Splunk documentation states that "The eval-expression can reference fields that are returned by the streamstats command. "

 

However it is completely ignoring my reset_after clause, any help is much appreciated. Example below:

 

 

index=events
|table id,_time

|sort 0 -id, _time

|streamstats current=f reset_on_change=true last(_time) as last_seen by id

|eval time_delta=_time-last_seen

|sort 0 -id, _time

|streamstats reset_after="("running_time>=1000")" reset_on_change=true sum(time_delta) as running_time by id

 

Splunk reference:

reset_afterSyntax: reset_after="("<eval-expression>")"Description: After the streamstats calculations are produced for an event, reset_after specifies that all of the accumulated statistics are reset if the eval-expression returns true. The eval-expression must evaluate to true or false. The eval-expression can reference fields that are returned by the streamstats command. When the reset_after argument is combined with the window argument, the window is also reset when the accumulated statistics are reset.

Labels (1)
0 Karma

patientsplunker
Loves-to-Learn Everything

Are you implying the first streamstats pipe is interfering with the second stremstats pipe ?

The output of the first streamstats is exactly what I expect, and i need the current=f so because time_delta is the difference between the time of an event and the event before it.

0 Karma

ITWhisperer
SplunkTrust
SplunkTrust

Sorry, that was me misreading your code. Sometimes it is not easy to distinguish code from prose. It usually makes it clearer by including code in a code block</>

 

0 Karma

patientsplunker
Loves-to-Learn Everything

Updated to code block 🙂 

0 Karma

ITWhisperer
SplunkTrust
SplunkTrust

reset_on_change is overriding reset_after - reset_on_change operates on the value of the field(s) in the by clause

0 Karma

patientsplunker
Loves-to-Learn Everything

Thank you for the response!

 

I removed the reset_on_change clause, however it is still ignoring my reset_after, the aggregate "running_time" grows past 1000 and keeps going for all rows. The reason I included the reset_on_change is because i also need it to reset after the id changes.


0 Karma

ITWhisperer
SplunkTrust
SplunkTrust

It looks like the current=f is what is causing the issue - do you need that? Or can you subtract the time_delta from the running_time afterwards?

0 Karma

patientsplunker
Loves-to-Learn Everything

Continued thanks for the help

 

The current=f in the first streamstats pipe lets me determine the difference between an event the immediate preceeding event (time_delta), which is why i do the sort. The output of the first streamstats is exactly what I expect, and the second streamstats does not have current=f, so i'm confused if there is interplay between the two streamstats

0 Karma

ITWhisperer
SplunkTrust
SplunkTrust

I don't think you need the reset_on_change=t in the second streamstats because you have current=f in the first streamstats making time_delta null which effectively resets running_time.

However, this doesn't explain why reset_after isn't working - looks like a bug?

0 Karma

ITWhisperer
SplunkTrust
SplunkTrust

Definitely, looks like a bug. Here is a runanywhere example demonstrating the issue (although it uses random ids so may not show it every time).

| gentimes start=-1 increment=1h
| rename starttime as _time 
| fields _time
| eval id=random()%3
| sort 0 -id _time
| streamstats current=f reset_on_change=t last(_time) as last_seen by id
| eval time_delta=_time-last_seen
| sort 0 -id _time
| streamstats reset_after="(sum(time_delta) > 10000)" sum(time_delta) by id
0 Karma

patientsplunker
Loves-to-Learn Everything

I appreciate the investigation, too bad it's a bug, essentially what im looking to accomplish is this, if you have any alternate ideas i would appreciate it.

starting dataset (this is after the sort)

_timeID
1001
10001
200001
220001
1002
4002
50002
59002
79002

 

desired output:

_timeIDlast_seentime_deltarunning_time
1001nullnull0
10001100900900
20000110001900019900 
220001200002000

2000

```this was reset since 1000 threshold crossed at 19900```

1002nullnull0 ```reset because the ID changed```
4002100300300
5000240046004900
6100250001100

1100

```this was reset since 1000 threshold crossed at 4900```

7900260001900

1900

```this was reset since 1000 threshold crossed at 1100```

 

0 Karma

ITWhisperer
SplunkTrust
SplunkTrust

OK it looks like it is because time_delta has a null value - try inserting this before the second streamstats

| fillnull value=0 time_delta
0 Karma

patientsplunker
Loves-to-Learn Everything

I think this is working!

0 Karma
Get Updates on the Splunk Community!

Prove Your Splunk Prowess at .conf25—No Prereqs Required!

Your Next Big Security Credential: No Prerequisites Needed We know you’ve got the skills, and now, earning the ...

Splunk Observability Cloud's AI Assistant in Action Series: Observability as Code

This is the sixth post in the Splunk Observability Cloud’s AI Assistant in Action series that digs into how to ...

Splunk Answers Content Calendar, July Edition I

Hello Community! Welcome to another month of Community Content Calendar series! For the month of July, we will ...