Best strategy to deal with / Fix Time zone (time_sych) issues between Splunk Servers & hosts scattered around the US?
How does one prepare a report of the Hosts having time zone differences & ways to fix the issues.
Splunk converts timestamps into a common time zone when they are received and the converts them to the user's preferred time zone at display time. If a timestamp is in the wrong zone then it probably was ingested incorrectly.
Ideally, the timestamp of an event will contain zone information. Failing that, the forwarder which sent the event can use the TZ setting in props.conf to specify the correct time zone.
Tell us more about the use case and we may be able to offer other solutions.
To fix the TZ issue. Do I need to check the props.com on every host? So if the host is in a specific TZ & it is correctly set, then no worries. Due to 100s of hosts. So the TZ on a host is set correctly , there are worries right? I thought I could save time by not having to log into every available Host containing a Forwarder.
Yes, to correct the TZ issue, the props.conf settings on each forwarder must be examined. If you have hundreds of hosts then you really should be using a Splunk deployment server (DS). The DS offers a single point for monitoring and managing forwarders. It saves tons of time by not having to sign in to each host to work on the forwarder.
See https://docs.splunk.com/Documentation/Splunk/8.1.2/Updating/Aboutdeploymentserver for more information about the DS.
I don't understand the question "So the TZ on a host is set correctly , there are worries right?". There are two time zone settings - the one for the host and the one for Splunk Universal Forwarder. We are concerned about the latter.