Splunk Search

Summary Index & sistats

Path Finder

Hi

I am seeing some weirdness with one of the saved-searches that we have. One of these searches is of the form:

... | bucket span=1h _time | sistats median(field1), avg(field1) by _time,field2,field3  

When I ran this query for the last 60 minutes, it generated a lot of results (close to 295K). When I ran the same query using the stats command instead, I got close to 160 results.

I am not sure why this is happening. We are using version 4.0.10. Thanks for your help.

Ranga

0 Karma
1 Solution

Splunk Employee
Splunk Employee

field1 is probably a numeric field with many distinct values. If you use sistats to calculate a median or other percentiles on such a field (or a distinct_count), you will wind up with a very large summary index, at least one entry for every distinct value. (In your case you probably have about 1600 distinct values, each accounting for about 160 entries.) This is correct, and is the only way to summarize a median in such a way that an aggregated median can be reduced from this.

If you don't want this, then you should first "bucket" field1. this will reduce the size of your summary, at some slight cost in the accuracy of aggregated medians.

View solution in original post

0 Karma

Splunk Employee
Splunk Employee

field1 is probably a numeric field with many distinct values. If you use sistats to calculate a median or other percentiles on such a field (or a distinct_count), you will wind up with a very large summary index, at least one entry for every distinct value. (In your case you probably have about 1600 distinct values, each accounting for about 160 entries.) This is correct, and is the only way to summarize a median in such a way that an aggregated median can be reduced from this.

If you don't want this, then you should first "bucket" field1. this will reduce the size of your summary, at some slight cost in the accuracy of aggregated medians.

View solution in original post

0 Karma

Splunk Employee
Splunk Employee

It occurs to me that you can use the function makecontinuous instead of bucket, and that should give you better results.

0 Karma

Splunk Employee
Splunk Employee

that's because bucket creates a range for non-time values, not a single value. you need to eval it apart into two numbers and decide what you want to represent the values within the range.

0 Karma

Path Finder

I tried adding bucket bins=50 field1 before the sistats command, but that results in the Median & Average values not getting calculated.

0 Karma
Don’t Miss Global Splunk
User Groups Week!

Free LIVE events worldwide 2/8-2/12
Connect, learn, and collect rad prizes
and swag!