topic Re: Collect bins time? in Splunk Search

Collect bins time?

larryp — Thu, 22 Jun 2017 01:58:07 GMT

OK, this is driving me crazy. I have a normal time in _time (displayed as yyyy-mm-dd HH:MM:SS). I collect it into an index without any bin. I search on the index, and the times are all yyyy-mm-dd HH:00:00 or yyyy-mm-dd HH:00:01. In addition, the number of events with the time yyyy-mm-dd HH:00:00 are various multiples of 100,000, as if there's a limitation being reached, but the multiple isn't the same.

I can copy the _time variable to another one, and it will be in the index exactly as I expect (displayed as epoch time with microsecond precision) from the index. But that AFAIK can't be used in, e.g., timechart. (And as this index is for users to run statistics on, copying it back to _time isn't an option.)

As I was researching this I came across a comment about sub-second times possibly creating a need for too much memory, but (a) I can't find it again, and (b) I would have thought that one would lose subsecond precision, not (in my case) hourly.

Any help would be appreciated!

Re: Collect bins time?

DalJeanis — Thu, 22 Jun 2017 15:23:51 GMT

Here's the apparent answer and workarounds...

https://answers.splunk.com/answers/218702/why-are-collected-events-in-a-summary-index-losing.html

If you do need the millisecond-level accuracy for timecharting, then you can always recalculate _time before the timechart command, either inline in the search or in props.conf the way @krdo suggests in the comments on that question.

Re: Collect bins time?

larryp — Thu, 22 Jun 2017 15:38:17 GMT

That's interesting, but apparently not the cause of my problem - I'm losing minute and second as well as millisecond granularity (where seconds are either 0 or 1). The work-around would only work for milliseconds.

Re: Collect bins time?

woodcock — Thu, 22 Jun 2017 20:06:07 GMT

There is a limit on the number of events that can exist at the same second. There is an error that is generated when this is hit.

https://answers.splunk.com/answers/303/whats-max-events-i-can-have-timestamped-with-a-particular-second-millisecond.html

Re: Collect bins time?

larryp — Thu, 22 Jun 2017 20:39:46 GMT

That's also interesting, but in my case why did Splunk ‘bin’ events into hourly buckets, where 100000 or 200000 events got put in second 0 (HH:00:00) and the rest (>100000) got put in second 1 (HH:00:01)? In fact, I don't want _time reduced to one arbitrary second out of 3600; I'd rather keep the events in the 3600 seconds for each hour they occur in.

I'm going to update the description of the problem with some clarifying information.

Re: Collect bins time?

larryp — Thu, 22 Jun 2017 20:45:26 GMT

Looks like I can't edit the question but only add new information via comment.

Turns out when I had search … | table … | collect … the _time gets collapsed to hourly increments (with rollover to subsequent seconds as @woodcock pointed out from https://answers.splunk.com/answers/303/whats-max-events-i-can-have-timestamped-with-a-particular-second-millisecond.html). but search … | fields … | collect … keeps _time with millisecond accuracy in the index.

Unfortunately table is only one requirement; stats is also desired.

Re: Collect bins time?

larryp — Fri, 23 Jun 2017 01:28:09 GMT

OK, I'm accepting this answer. It looks like the secondary effect is created by the ‘earliest’ unit, not the value or the span, nor any bin value (e.g, 5m).