Why is there Timing (parsing) Issue on Splunk Cloud?

[jmr44]

There  are five different hosts on our fleet on two different timezones with four sourcetypes on each. The problem is that the time that is being shown in Splunk Cloud isn't always the timestamp from the logs. They are different. 

The hosts pass the data through an intermediate forwarder (universal forwarder running inside) which is in UTC. 

There's also cases where one sourcetype from one host shows up/parses the correct time format but when they are coming from a different source, it doesn't.  I'll explain below:

Five different hosts - host_A (MST), host_B (MST), host_C (UTC), host_D (UTC), host_E (UTC)

Four different source types - src_W, src_X, src_Y, src_Z

For host_A (MST) and host_B (MST), src_W is shown at the correct time. src_X and src_Y are not. For example - if src_X and src_Y have the timestamp of 05/02/2022 14:xx:xx. in splunk, it shows as 04/02/2022 7:xx:xx. 
Between these two, src_Z only comes from host_A and the timestamp of 05/02/2022 14:xx:xx. in splunk, it shows as 04/02/2022 9:xx:xx. 

For host_C (UTC) - if src_W and src_X have the timestamp of 05/02/2022 21:xx:xx. in splunk, it shows as 04/02/2022 2:xx:xx. host_C doesn't have Y and Z. 

For host_D (UTC) - if src_Y has the timestamp of 05/02/2022 21:xx:xx. in splunk, it shows as 04/02/2022 2:xx:xx. host_D doesn't have the other sourcetypes.

For host_E (UTC) - if src_Y has the timestamp of 05/02/2022 21:xx:xx. in splunk, it shows as 04/02/2022 2:xx:xx. host_E doesn't have the other sourcetypes. For src_Z timestamp of 05/02/2022 14:xx:xx. in splunk, it shows as 04/02/2022 9:xx:xx - just like in host_A.

Sorry this might seem to be very complicated and it is in MST and not PST like I said before. My Splunk Cloud instance is also set to MST. 

Below is how the log formatting looks like:

This is how log from src_W is: 
eni=xx.yy.zz.aa client_ip=- - - [05/Feb/2023:17:46:53 -0700] ... ... ....

This is how log from  src_X is:
DEBUG 2023-02-06 00:49:22 ... ... ...

This is how log from src_Y is: 

INFO 2023-02-06 00:50:02 ... ... ...

This is how log from src_Z is:

qwertyui Sun Feb 5 04:40:39 2023:


Thank you for the help!

[shivanshu1593]

Well, for the timezone issues, you can always go to settings -> sourcetypes, find your sourcetype, click on edit and under advanced tab put the following. This will get pushed to your indexers and you would be good in that regard.

TZ = <Your preferred timezone>

Mountain time (MST) is 7 hours behind UTC. And you confirmed that the timezone set in your user' preferred timezone in Splunkcloud is also MST. The timestamp that eventually will show up as _time for you will always be MST. So for the log sources coming via servers in UTC, you'll always see _time being 7 hours behind the time you see in the event. That's not going away. If you want to see _time match with the time present in the events coming from servers in UTC, you need to change your preferred timezone to UTC. Splunk respects the time present in the events, so you'll always see the differences for obvious reasons.

 What you can also do is create a props.conf and deploy it to all your servers with your preferred TZ. That will be get applied as metadata and when the data eventually reaches the indexers, you'll always see the _time as your preferred timezone. For the events coming from UTC, you'll see them 7 hours ahead in the events, but _time will always be MST. And as always, the last pass to resolve all of this is to either sync all servers with NTP or set them into the same timezones, which obviously may not be feasible but just putting it out there.