The short answer is yes: the best practices are to make sure that Network Time Protocol (NTP) is configured, and to set all Splunk infrastructure to Universal Time Coordinated (UTC). But let's unpack this to understand why and when they apply.
Network Time Protocol (NTP) is a global service used to synchronize digital clocks. Smartphones are always right because they synchronize regularly with NTP. Analog clocks don't connect to NTP, so they usually drift fast or slow.
A general best practice is to configure your whole infrastructure to use NTP, not just your Splunk deployment. You can configure NTP to synchronize to either a public or private (internal) NTP server.
Using NTP for your Splunk infrastructure becomes more critical as the deployment becomes more distributed. Many internal Splunk Enterprise communications depend on accurate times, such as bundle pushes, cluster synchronizations, and replication factors. Also, troubleshooting across Splunk Enterprise components can be difficult if the timestamps are not synchronized to NTP.
UTC is the time standard commonly used to keep the world's timing centers closely synchronized. The UTC standard uses the same current time as the GMT time zone.
As with any distributed technology, managing a distributed Splunk deployment can be unnecessarily complicated if the underlying hardware is set in different time zones. Although Splunk software will provide time-accurate data storage, search, and analytics no matter what time zone the indexers and search heads are set to, having different time zones on different Splunk Enterprise servers can cause confusion.
For example, say your Splunk infrastructure is hosted in data centers on the east and west coasts of the USA. If you configure the servers with their own local time zones, the various servers in your Splunk infrastructure will have different clock times. If you configure all the servers in both locations with the local time of one location, the servers in the other location will be set to a time zone they are not physically in. Either scenario can cause confusion.
As a best practice, set all the servers in your Splunk deployment infrastructure to the UTC time zone, even if the servers reside in different physical time zones. All the servers in your infrastructure will show the same time, so it can't be confused with the local time of the observer. An exception to this would be if your IT organization has a firm policy for using a different time zone scheme.
In general, the Splunk platform will store data correctly, and provide accurate search and analysis regardless of what time zone the servers that host Splunk Enterprise use, because most data bears the timestamp it received when its underlying event occurred.
If an event doesn't have a timestamp, though, or the timestamp appears to be too far in the future, or too far in the past, Splunk Enterprise uses the time that it indexes your data as the timestamp for the event. That's why synchronization and time zone assignment are important for search and analysis. To learn more, see How time zones are processed by the Splunk platform in the Search Manual.
In the Splunk UI, search results are displayed in the time zone specified for the user doing the searching. This display time zone does not change the timestamp of the underlying event data, whose time zone is determined at event-creation time, or at index time.
For details about how time zones are used in timestamps, and how a user can set the time zone for their search results, see Specify time zones for time stamps in the Getting Data In Manual.
All currently supported forwarders communicate the forwarder host's time zone in their forwarding streams. To keep timestamps as accurate as possible at index and search time, you should make sure that the data source includes its time zone in the event timestamp, for example,
2019-10-10 3:45PM EST, not just
2019-10-10 3:45PM. If that's not possible, you can define the data source's time zone using source types.
For example, say a host system is set to eastern standard time (UTC-5), but a business application produces logs in UTC and does not include the time zone in the event timestamp. If the data source's time zone is not configured in props.conf, when Splunk software processes the log file in UTC-5, it will assume the timestamp to be that of the local forwarder host, resulting in data five hours behind reality.
To learn more about using the source type to define time zones, see Specify time zones for timestamps in the Splunk Enterprise Getting Data In manual.