topic Re: Why are streamstats reset_after for streamstats aggregation not working? in Splunk Search

Why are streamstats reset_after for streamstats aggregation not working?

patientsplunker — Fri, 19 May 2023 14:33:42 GMT

Hello,

I am trying to use Streamstats with Sum(value) and I want to reset that sum after it reaches a certain threshold (in the example below this threshold is 1000). The Splunk documentation states that "The eval-expression can reference fields that are returned by the streamstats command. "

However it is completely ignoring my reset_after clause, any help is much appreciated. Example below:

Splunk reference:

reset_afterSyntax: reset_after="("<eval-expression>")"Description: After the streamstats calculations are produced for an event, reset_after specifies that all of the accumulated statistics are reset if the eval-expression returns true. The eval-expression must evaluate to true or false. The eval-expression can reference fields that are returned by the streamstats command. When the reset_after argument is combined with the window argument, the window is also reset when the accumulated statistics are reset.

Re: Why are streamstats reset_after for streamstats aggregation not working?

ITWhisperer — Fri, 19 May 2023 13:49:00 GMT

reset_on_change is overriding reset_after - reset_on_change operates on the value of the field(s) in the by clause

Re: Why are streamstats reset_after for streamstats aggregation not working?

patientsplunker — Fri, 19 May 2023 13:58:07 GMT

Thank you for the response!

I removed the reset_on_change clause, however it is still ignoring my reset_after, the aggregate "running_time" grows past 1000 and keeps going for all rows. The reason I included the reset_on_change is because i also need it to reset after the id changes.

Re: Why are streamstats reset_after for streamstats aggregation not working?

ITWhisperer — Fri, 19 May 2023 14:07:13 GMT

It looks like the current=f is what is causing the issue - do you need that? Or can you subtract the time_delta from the running_time afterwards?

Re: Why are streamstats reset_after for streamstats aggregation not working?

patientsplunker — Fri, 19 May 2023 14:17:15 GMT

Are you implying the first streamstats pipe is interfering with the second stremstats pipe ?

The output of the first streamstats is exactly what I expect, and i need the current=f so because time_delta is the difference between the time of an event and the event before it.

Re: Why are streamstats reset_after for streamstats aggregation not working?

patientsplunker — Fri, 19 May 2023 14:24:37 GMT

Continued thanks for the help

The current=f in the first streamstats pipe lets me determine the difference between an event the immediate preceeding event (time_delta), which is why i do the sort. The output of the first streamstats is exactly what I expect, and the second streamstats does not have current=f, so i'm confused if there is interplay between the two streamstats

Re: Why are streamstats reset_after for streamstats aggregation not working?

ITWhisperer — Fri, 19 May 2023 14:31:19 GMT

Sorry, that was me misreading your code. Sometimes it is not easy to distinguish code from prose. It usually makes it clearer by including code in a code block</>

Re: Why are streamstats reset_after for streamstats aggregation not working?

patientsplunker — Fri, 19 May 2023 14:35:18 GMT

Updated to code block 🙂

Re: Why are streamstats reset_after for streamstats aggregation not working?

ITWhisperer — Fri, 19 May 2023 14:39:45 GMT

I don't think you need the reset_on_change=t in the second streamstats because you have current=f in the first streamstats making time_delta null which effectively resets running_time.

However, this doesn't explain why reset_after isn't working - looks like a bug?

Re: Why are streamstats reset_after for streamstats aggregation not working?

ITWhisperer — Fri, 19 May 2023 14:52:55 GMT

Definitely, looks like a bug. Here is a runanywhere example demonstrating the issue (although it uses random ids so may not show it every time).

Re: Why are streamstats reset_after for streamstats aggregation not working?

ITWhisperer — Fri, 19 May 2023 15:20:02 GMT

OK it looks like it is because time_delta has a null value - try inserting this before the second streamstats

| fillnull value=0 time_delta

Re: Why are streamstats reset_after for streamstats aggregation not working?

patientsplunker — Fri, 19 May 2023 15:23:05 GMT

I appreciate the investigation, too bad it's a bug, essentially what im looking to accomplish is this, if you have any alternate ideas i would appreciate it.

starting dataset (this is after the sort)

_time	ID
100	1
1000	1
20000	1
22000	1
100	2
400	2
5000	2
5900	2
7900	2

desired output:

_time	ID	last_seen	time_delta	running_time
100	1	null	null	0
1000	1	100	900	900
20000	1	1000	19000	19900
22000	1	20000	2000	2000 ```this was reset since 1000 threshold crossed at 19900```
100	2	null	null	0 ```reset because the ID changed```
400	2	100	300	300
5000	2	400	4600	4900
6100	2	5000	1100	1100 ```this was reset since 1000 threshold crossed at 4900```
7900	2	6000	1900	1900 ```this was reset since 1000 threshold crossed at 1100```

Re: Why are streamstats reset_after for streamstats aggregation not working?

patientsplunker — Fri, 19 May 2023 15:51:56 GMT

I think this is working!