Fix timezone Issue - logs showing up in UTC time but generated in a different time

[jmr44]

Background:
I am sending data to Splunk Cloud through an Intermediate Forwarder, which is a universal forwarder from multiple source instances (both in Pacific time and UTC) that do not support HEC and the correct TLS versions. As of now, this is the only way of sending the logs. 
Sources (PT and UTC) > Intermediate Forwarder > Splunk Cloud

Problem: 

Some of the source instances are in Pacific time zone and the intermediate forwarder is in UTC. The logs that are coming from the instances that are in PT, are showing up in UTC time in Splunk Cloud. So Splunk Cloud is showing that the logs are from a time earlier than the they really got generated. 

I should also say that the logs have a timestamp (PT - correct time) in them, always within the first 70 characters or so at most. That is the time I want them to be shown in. 
Also changing the intermediate forwarder timezone to PT fixes the issue for the instances in PT but messes up the instances that are in UTC. 

As a solution, I saw that timezones can be configured in the props.conf file, located in (after creation) /opt/splunkforwarder/etc/system/local directory. 
Here is what it looks like for me: 

[host::<some of the hosts>]

TZ = US/Pacific

I have tried adding the following argument in the stanza (according to this post https://community.splunk.com/t5/Getting-Data-In/Universal-Forwarder-and-props-conf-and-transforms-co...) but it is of no help. 

force_local_processing = true

 However, this just does not work. I can see logs in Splunk Cloud but they are in UTC. 

I have also tried using <sourcetype> and source::<source> instead of [host::<myhost>] but it doesn't do anything. 
Any other things that I can try?

[richgalloway]

What you've tried should work if you use the right time zone value.  Try either the canonical name

TZ = America/Los_Angeles

 or the abbreviation

TZ = PST

[jmr44]

Hi @richgalloway , thank you so much for your reply!
TZ = America/Los_Angeles has helped me a bit. 
However, I am not joking but the logs of  only one sourcetype of <myhost> show up. The other 3 don't. 
This is what I have now:

[host::<my host>]
TZ = America/Los_Angeles
force_local_processing = true

Any suggestions? 

[richgalloway]

Don't use force_local_processing.  It doesn't solve your problem.

When you say 3 sourcetypes don't show up do you mean they don't have the correct time zone or they aren't present at all?  If the latter then you have a different problem that should be in a new question.

If the former then try putting the TZ setting in sourcetype: stanzas rather than host: stanzas.

[jmr44]

@richgalloway , all of them show up but Splunk shows a different time, than the time posted in the logs. 

The timestamp is in the logs within the first 50-60 characters but Splunk shows them at a different time. 

On a sidenote, can I filter out the logs by combining both host and sourcetype? For example: 
[host::<myhost> and <sourcetype>]? 

Now I am just doing [host::<myhost>]

For example, in the log:

INFO 2023-02-04 20:47:59 XXXYYYZZZ AAA BBB CCC - that is when it is generated and sent

but splunk shows: 2023-02-04T13:47. 

[richgalloway]

No, you cannot combine host and sourcetype in a stanza name.

Do all 3 of the incorrect sourcetypes have the same format?  If so, then they should be a single sourcetype.  Otherwise, please share the others so we can better figure out why times are not interpreted correctly.

Confirm you restarted the UF after changing props.conf.

[jmr44]

@richgalloway I have restarted. 

They do not. There's also cases where one sourcetype from one host shows up/parses the correct time format but when they are coming from a different source, it doesn't.  I'll explain below:

Five different hosts - host_A (MST), host_B (MST), host_C (UTC), host_D (UTC), host_E (UTC)

Four different source types - src_W, src_X, src_Y, src_Z

For host_A (MST) and host_B (MST), src_W is shown at the correct time. src_X and src_Y are not. For example - if src_X and src_Y have the timestamp of 05/02/2022 14:xx:xx. in splunk, it shows as 04/02/2022 7:xx:xx. 
Between these two, src_Z only comes from host_A and the timestamp of 05/02/2022 14:xx:xx. in splunk, it shows as 04/02/2022 9:xx:xx. 

For host_C (UTC) - if src_W and src_X have the timestamp of 05/02/2022 21:xx:xx. in splunk, it shows as 04/02/2022 2:xx:xx. host_C doesn't have Y and Z. 

For host_D (UTC) - if src_Y has the timestamp of 05/02/2022 21:xx:xx. in splunk, it shows as 04/02/2022 2:xx:xx. host_D doesn't have the other sourcetypes.

For host_E (UTC) - if src_Y has the timestamp of 05/02/2022 21:xx:xx. in splunk, it shows as 04/02/2022 2:xx:xx. host_E doesn't have the other sourcetypes. For src_Z timestamp of 05/02/2022 14:xx:xx. in splunk, it shows as 04/02/2022 9:xx:xx - just like in host_A.

Sorry this might seem to be very complicated and it is in MST and not PST like I said before. My Splunk Cloud instance is also set to MST. 

Below is how the log formatting looks like:

This is how log from src_W is: 
eni=xx.yy.zz.aa client_ip=- - - [05/Feb/2023:17:46:53 -0700] ... ... ....

This is how log from  src_X is:
DEBUG 2023-02-06 00:49:22 ... ... ...

This is how log from src_Y is: 

INFO 2023-02-06 00:50:02 ... ... ...

This is how log from src_Z is:

qwertyui Sun Feb 5 04:40:39 2023:


Thank you for the help!

[richgalloway]

Thanks for the details.

I'll start with the easy one.  I see that src_W contains time zone information in the timestamp.  This means Splunk should have no difficulty setting the right time, provided the TIME_FORMAT setting is correct (%d/%B/%Y:%H:%M:%S%z) for the sourcetype.

src_X and src_Y have the same time format, but that's different from src_W.  src_Z has yet another time format.  Verify each source references a sourcetype that has the appropriate TIME_FORMAT setting.  

If I haven't said it before, I'll say it now.  Make sure you're looking at a real problem.  Splunk will display times in your selected time zone so something you may expect to see in UTC will appear in MST.  If you change your selected time zone to UTC and times still are off by hours then you have a real problem.