Getting Data In

Configuring to support searches for events timestamped in future

matt_1
Explorer

We have an global application hosted within a VM environment feeding a common Splunk index server. However the server times set onto these VMs are relative to the region they support (EMEA hosts set to CET/CEST, APAC set to SSG). The Splunk index and distributed search servers themselves are in CST/CDT. The logs from all regions will contain timestamps relative to their UTC offset.

How would I get around the issue where data coming in +8h to +14h CST/CDT has be be searched in the future? Ideally, I would like the searches used by our APAC and EMEA users to still resolve in their local time, but I'd also like to insure the cross-region correlation is not impacted by disparities caused by UTC offsets.

Is this an instance where the TZ needs to be specified at the source (ie light forwarders); or is this something we define at search or index time? Anything has to be better than what we are doing now, which is setting the offsets within times.conf for the EMEA and APAC search heads.

Tags (1)
0 Karma
1 Solution

parallaxed
Path Finder

AFAIK, search nodes can have their timezones adjusted using the TZ environment variable, but the offset/timestamping will officially occur at index time, so you should configure it on your indexing heads.

You can define it using "TZ" in props, providided you have some distinguishing feature (perhaps in the hostname or source path) available to you, e.g.:

[host::apac*] TZ=Asia/Tokyo

[source::/path/to/apac*] TZ=Asia/Tokyo

http://docs.splunk.com/Documentation/Splunk/5.0/Admin/Propsconf

If your data is somehow farther in the future (forecast data or what have you), you'll have to up the MAX_DAYS_HENCE value from it's default of 2.

View solution in original post

0 Karma

parallaxed
Path Finder

AFAIK, search nodes can have their timezones adjusted using the TZ environment variable, but the offset/timestamping will officially occur at index time, so you should configure it on your indexing heads.

You can define it using "TZ" in props, providided you have some distinguishing feature (perhaps in the hostname or source path) available to you, e.g.:

[host::apac*] TZ=Asia/Tokyo

[source::/path/to/apac*] TZ=Asia/Tokyo

http://docs.splunk.com/Documentation/Splunk/5.0/Admin/Propsconf

If your data is somehow farther in the future (forecast data or what have you), you'll have to up the MAX_DAYS_HENCE value from it's default of 2.

0 Karma

matt_1
Explorer

Thanks. I suspected so, but was hoping there was another method. Unfortunately we don't have any distinguishing elements outside of the sourcetypes we're setting. I'll place an enhancement request to at least allow simple pattern matching for sourcetypes within props.

0 Karma
Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

Splunk is officially part of Cisco

Revolutionizing how our customers build resilience across their entire digital footprint.   Splunk ...

Splunk APM & RUM | Planned Maintenance March 26 - March 28, 2024

There will be planned maintenance for Splunk APM and RUM between March 26, 2024 and March 28, 2024 as ...