There are five different hosts on our fleet on two different timezones with four sourcetypes on each. The problem is that the time that is being shown in Splunk Cloud isn't always the timestamp from the logs. They are different.
The hosts pass the data through an intermediate forwarder (universal forwarder running inside) which is in UTC.
There's also cases where one sourcetype from one host shows up/parses the correct time format but when they are coming from a different source, it doesn't. I'll explain below:
Five different hosts - host_A (MST), host_B (MST), host_C (UTC), host_D (UTC), host_E (UTC)
Four different source types - src_W, src_X, src_Y, src_Z
For host_A (MST) and host_B (MST), src_W is shown at the correct time. src_X and src_Y are not. For example - if src_X and src_Y have the timestamp of 05/02/2022 14:xx:xx. in splunk, it shows as 04/02/2022 7:xx:xx. Between these two, src_Z only comes from host_A and the timestamp of 05/02/2022 14:xx:xx. in splunk, it shows as 04/02/2022 9:xx:xx.
For host_C (UTC) - if src_W and src_X have the timestamp of 05/02/2022 21:xx:xx. in splunk, it shows as 04/02/2022 2:xx:xx. host_C doesn't have Y and Z.
For host_D (UTC) - if src_Y has the timestamp of 05/02/2022 21:xx:xx. in splunk, it shows as 04/02/2022 2:xx:xx. host_D doesn't have the other sourcetypes.
For host_E (UTC) - if src_Y has the timestamp of 05/02/2022 21:xx:xx. in splunk, it shows as 04/02/2022 2:xx:xx. host_E doesn't have the other sourcetypes. For src_Z timestamp of 05/02/2022 14:xx:xx. in splunk, it shows as 04/02/2022 9:xx:xx - just like in host_A.
Sorry this might seem to be very complicated and it is in MST and not PST like I said before. My Splunk Cloud instance is also set to MST.
Below is how the log formatting looks like:
This is how log from src_W is: eni=xx.yy.zz.aa client_ip=- - - [05/Feb/2023:17:46:53 -0700] ... ... ....
This is how log from src_X is: DEBUG 2023-02-06 00:49:22 ... ... ...
This is how log from src_Y is:
INFO 2023-02-06 00:50:02 ... ... ...
This is how log from src_Z is:
qwertyui Sun Feb 5 04:40:39 2023: Thank you for the help!
... View more