topic Re: Why is bin command creating too many bins? Issue with 5-minute segments in Deployment Architecture

Why is bin command creating too many bins? Issue with 5-minute segments

FeatureCreeep — Fri, 03 Nov 2017 16:34:45 GMT

I have the same problem that is unanswered here

I'm trying to do stats on the last 10 minutes of data by two separate 5 minute buckets.

My command looks like this:

host=servers* index=iis | bin _time bins=2 | stats count as Request by _time, host

When I run this for the last 10 minutes I expect 2 separate 5 minute buckets. If my query starts on an even 5 minute interval like 1:05-1:15, it works fine but the buckets expect even 5 minute increments and if I run 1:02-1:12, it will create a "1:00" bucket for the requests between 1:00 and 1:05, a "1:05" bucket for the 1:05-1:10 requests, and a "1:10" bucket for the 1:10-1:15 requests. That is 3 buckets of different sizes.

Using the span=5 minutes doesn't help either.

I tried just using the epoch number like:

eval Time=_time | bin Time bins=2

I was trying to stop Splunk from treating the _time field specially but it put everything into a single bin.

How can I get Splunk to just create 2 evenly sized/spanned buckets?

Re: Why is bin command creating too many bins? Issue with 5-minute segments

jgbricker — Fri, 03 Nov 2017 17:33:33 GMT

can you clarify what isn't working about |bin span=5min _time You may also find |convert ctime(_time) as time useful because sometimes it comes out as epoch format. convert has a 'timeformat=' property also to help output in preferred syntax

btw.. bins doesn't force the number of bins it sets a limit

binsSyntax: bins=Description: Sets the maximum number of bins to discretize into.
https://docs.splunk.com/Documentation/SplunkCloud/6.6.3/SearchReference/Bin

Re: Why is bin command creating too many bins? Issue with 5-minute segments

FeatureCreeep — Fri, 03 Nov 2017 17:51:05 GMT

It won't create just 2 bins. Notice that the query goes back 10 minutes and the span is 5 minutes, but I still get 3 buckets. It's the same if I use "bins=2"

This query: host=server* index=iis | bin span=5min _time | stats count by _time, host

Run for this time: (11/3/17 1:37:17.000 PM to 11/3/17 1:47:17.000 PM)

Returns these results:

_time host count
2017-11-03 13:35:00 server01 7339
2017-11-03 13:40:00 server01 12910
2017-11-03 13:45:00 server01 6432
2017-11-03 13:35:00 server02 7402
2017-11-03 13:40:00 server02 14509
2017-11-03 13:45:00 server02 6167
2017-11-03 13:35:00 server03 7034
2017-11-03 13:40:00 server03 13665
2017-11-03 13:45:00 server03 6273

Re: Why is bin command creating too many bins? Issue with 5-minute segments

somesoni2 — Fri, 03 Nov 2017 18:13:35 GMT

Splunk's bucketing, on time, is done considering starting point as 01/01/1970 00:00, hence the bucketing of 5 mins always takes minutes with multiplicating factor of 5 (0,5,10,15...).

If you just want to two buckets with last 5 mins to now and last 10 mins to last 5 mins, you need to use other workarounds, like this

host=server* index=iis | addinfo| eval _time=if(_time<relative_time(info_min_time,"-5m@m"),relative_time(info_max_time,"@m"),relative_time(info_min_time,"-5m@m"))| stats count by _time, host

Re: Why is bin command creating too many bins? Issue with 5-minute segments

FeatureCreeep — Fri, 03 Nov 2017 19:49:56 GMT

Perfect! This information should be in the "bin" documentation. Your workaround strategy worked great though a few min and maxes were backwards that I corrected. Thanks!

Here is the updated workaround:

host=server* index=iis | addinfo| eval _time=if(_time<relative_time(info_max_time,"-5m@m"),relative_time(info_min_time,"@m"),relative_time(info_max_time,"-5m@m"))  | stats count by _time, host