Splunk Search

Will _time of raw events in an index change in the summary index if searches to summarize data are run using an admin account set to a different timezone?

DanielFordWA
Contributor

Hi,

I have raw events in an index that are set to universal time.

props.conf

[iis-prod]
TZ = Universal

The admin account is used to create searches off the raw index and summarize the data to speed up reporting.

index=iis publicationId=* | bucket _time span=1d | stats count by _time GUID publicationId DocAction

I set up a search to run every day and add this to a summary index.

The admin account is set to GMT London time zone.

Will this change the _time in the raw event and index the new _time as BST in the summary index?

I am seeing a slight difference in data between the raw and summarized indexes.

Hope you can help?

Thanks,

Dan

0 Karma
1 Solution

woodcock
Esteemed Legend

Yes, and here is why. I assume that your saved search is set to run every day just after midnight with earliest=-2d@d and latest=-1d@d (or similar). But what (or rather, "when") is a "day" (as regards the snap-to part)? If your user is in TZ=BST, then the window that defines "all of yesterday" is several hours different than the window that would be defined if he had TZ=UTC. If you were using hourly SIs, you would (probably, unless you are in North Korea or some other not-an-even-hour TZ delta) not be having this problem and maybe that is what you should be doing instead of rolling up daily.

View solution in original post

woodcock
Esteemed Legend

Yes, and here is why. I assume that your saved search is set to run every day just after midnight with earliest=-2d@d and latest=-1d@d (or similar). But what (or rather, "when") is a "day" (as regards the snap-to part)? If your user is in TZ=BST, then the window that defines "all of yesterday" is several hours different than the window that would be defined if he had TZ=UTC. If you were using hourly SIs, you would (probably, unless you are in North Korea or some other not-an-even-hour TZ delta) not be having this problem and maybe that is what you should be doing instead of rolling up daily.

DanielFordWA
Contributor

Hi,

Thanks for your reply.

I have experimented with pulling the events by second into the summary index.

index=iis publicationId=* | stats count by _time GUID publicationId DocAction

however I have the same issue, that while totals may add up over the entire index the days do not match the raw index when performing the search.

index=dbuserdoc earliest=-151d@d latest=-100d@d | bucket _time span=1d | stats sum(count) by _time GUID publicationId DocAction

index=iis earliest=-151d@d latest=-100d@d | bucket _time span=1d | stats count by _time GUID publicationId DocAction

Is this because I have to do a number of summary backloads and specify the earliest and latest as -100d@d -75d@d and -75d@d -50d@d etc?

If so how do I overcome the back load issue?

0 Karma

woodcock
Esteemed Legend

It makes no sense to do a Summary Index by seconds unless you are testing out some theory. If you are running all of your SI-populating searches AND your test searches under the same user (so the same TZ setting is being applied to the concept of "day", then they should line up. if they do not, the problem is likely that you are not accounting for pipeline lag and are doing your SI-populating searches "too soon" in that they are doing the summary for "last hour" before all the events that will eventually be in "last hour" have been indexed. This is why the documentation strongly advises doing your hourly SI-populating search from -2h@h to -1h@h instead of from -1h@h to 0h@h.

0 Karma

DanielFordWA
Contributor

Thank you for the reply and for the information regarding the time delay. I have discovered the difference is down to an error in a testers lookup across apps.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...