Configuring to support searches for events timestamped in future

[matt_1]

We have an global application hosted within a VM environment feeding a common Splunk index server. However the server times set onto these VMs are relative to the region they support (EMEA hosts set to CET/CEST, APAC set to SSG). The Splunk index and distributed search servers themselves are in CST/CDT. The logs from all regions will contain timestamps relative to their UTC offset.

How would I get around the issue where data coming in +8h to +14h CST/CDT has be be searched in the future? Ideally, I would like the searches used by our APAC and EMEA users to still resolve in their local time, but I'd also like to insure the cross-region correlation is not impacted by disparities caused by UTC offsets.

Is this an instance where the TZ needs to be specified at the source (ie light forwarders); or is this something we define at search or index time? Anything has to be better than what we are doing now, which is setting the offsets within times.conf for the EMEA and APAC search heads.

[parallaxed · Accepted solution]

AFAIK, search nodes can have their timezones adjusted using the TZ environment variable, but the offset/timestamping will officially occur at index time, so you should configure it on your indexing heads.

You can define it using "TZ" in props, providided you have some distinguishing feature (perhaps in the hostname or source path) available to you, e.g.:

[host::apac*] TZ=Asia/Tokyo

[source::/path/to/apac*] TZ=Asia/Tokyo

http://docs.splunk.com/Documentation/Splunk/5.0/Admin/Propsconf

If your data is somehow farther in the future (forecast data or what have you), you'll have to up the MAX_DAYS_HENCE value from it's default of 2.

[matt_1]

Thanks. I suspected so, but was hoping there was another method. Unfortunately we don't have any distinguishing elements outside of the sourcetypes we're setting. I'll place an enhancement request to at least allow simple pattern matching for sourcetypes within props.