Background: We have a system that allows salespersons to input the datetime of a salescall. This system is being used globally, and the system does not convert or normalize the user input to a standardized timezone after the user enters the datetime. Which means to say if the user indicates that a salescall was made at 10am on 28 Sep 2018, this value is stored in the DB and assumed to be relative to the user's local timezone
The problem: Splunk appears to be normalizing user-entered datetime fields to the UTC timezone. Our DB/Mysql server is currently based in Singapore (GMT+8) and when we performed a search query of all the datetime fields in the index, their values appears to be converted from their original DB values with -8h offset (including user entered fields).
Attempts at resolution: Based on what we could gather from the web (some links below), we have tried to do the following resolutions, either on their own or a combination: changing the timezone of the DBX connection to UTC from Asia/Singapore, changing the user time zone from default to (GMT+8), and also adding timezone settings ([host:: ] TZ: UTC; we have tried both UTC and Asia/Singapore) in the props.conf file. None seemed to work.
http://docs.splunk.com/Documentation/Splunk/7.1.2/Data/Applytimezoneoffsetstotimestamps
https://answers.splunk.com/answers/135193/splunk-indexing-and-time-zone-normalization.html
https://answers.splunk.com/answers/439363/how-to-set-a-default-timezone-for-an-entire-multis.html
Appreciate advice on how we can get Splunk to retain the original datetime values without any conversion or normalization when indexing. Disabling Splunk's default normalization of timezone across all datetime values (regardless of whether they are user-entered) would probably be the most straight-forward resolution we can think of.
