topic Re: Will _time of raw events in an index change in the summary index if searches to summarize data are run using an admin account set to a different timezone? in Splunk Search

Will _time of raw events in an index change in the summary index if searches to summarize data are run using an admin account set to a different timezone?

DanielFordWA — Mon, 10 Aug 2015 13:10:18 GMT

Hi,

I have raw events in an index that are set to universal time.

props.conf

[iis-prod]
TZ = Universal

The admin account is used to create searches off the raw index and summarize the data to speed up reporting.

index=iis publicationId=* | bucket _time span=1d | stats count by _time GUID publicationId DocAction

I set up a search to run every day and add this to a summary index.

The admin account is set to GMT London time zone.

Will this change the _time in the raw event and index the new _time as BST in the summary index?

I am seeing a slight difference in data between the raw and summarized indexes.

Hope you can help?

Thanks,

Dan

Re: Will _time of raw events in an index change in the summary index if searches to summarize data are run using an admin account set to a different timezone?

woodcock — Mon, 10 Aug 2015 14:26:37 GMT

Yes, and here is why. I assume that your saved search is set to run every day just after midnight with earliest=-2d@d and latest=-1d@d (or similar). But what (or rather, "when") is a "day" (as regards the snap-to part)? If your user is in TZ=BST, then the window that defines "all of yesterday" is several hours different than the window that would be defined if he had TZ=UTC. If you were using hourly SIs, you would (probably, unless you are in North Korea or some other not-an-even-hour TZ delta) not be having this problem and maybe that is what you should be doing instead of rolling up daily.

Re: Will _time of raw events in an index change in the summary index if searches to summarize data are run using an admin account set to a different timezone?

DanielFordWA — Tue, 11 Aug 2015 11:10:26 GMT

Hi,

Thanks for your reply.

I have experimented with pulling the events by second into the summary index.

index=iis publicationId=* | stats count by _time GUID publicationId DocAction

however I have the same issue, that while totals may add up over the entire index the days do not match the raw index when performing the search.

index=dbuserdoc earliest=-151d@d latest=-100d@d | bucket _time span=1d | stats sum(count) by _time GUID publicationId DocAction

index=iis earliest=-151d@d latest=-100d@d | bucket _time span=1d | stats count by _time GUID publicationId DocAction

Is this because I have to do a number of summary backloads and specify the earliest and latest as -100d@d -75d@d and -75d@d -50d@d etc?

If so how do I overcome the back load issue?

Re: Will _time of raw events in an index change in the summary index if searches to summarize data are run using an admin account set to a different timezone?

woodcock — Fri, 14 Aug 2015 01:09:51 GMT

It makes no sense to do a Summary Index by seconds unless you are testing out some theory. If you are running all of your SI-populating searches AND your test searches under the same user (so the same TZ setting is being applied to the concept of "day", then they should line up. if they do not, the problem is likely that you are not accounting for pipeline lag and are doing your SI-populating searches "too soon" in that they are doing the summary for "last hour" before all the events that will eventually be in "last hour" have been indexed. This is why the documentation strongly advises doing your hourly SI-populating search from -2h@h to -1h@h instead of from -1h@h to 0h@h.

Re: Will _time of raw events in an index change in the summary index if searches to summarize data are run using an admin account set to a different timezone?

DanielFordWA — Fri, 14 Aug 2015 08:18:55 GMT

Thank you for the reply and for the information regarding the time delay. I have discovered the difference is down to an error in a testers lookup across apps.