topic Re: Why is Splunk Universal Forwarder timezone incorrect after daylight saving time change? in Getting Data In

Why is Splunk Universal Forwarder timezone incorrect after daylight saving time change?

mattbg — Mon, 04 Apr 2022 16:37:31 GMT

Using Splunk UF 8.1.1, we've noticed an issue where the Linux x64 forwarder running on RedHat 7.7 did not seem to correctly adjust for daylight saving time; that is, the timestamps after the DST change are 1 hour ahead of where they should be.

We are not using any special TZ configuration on the UF or indexer and have until now relied on the Splunk UF picking up the underlying OS timezone to enrich events which, as I understand from the props.conf spec, is a supported approach.

Simply restarting the UF has resolved the issue on multiple servers.

The same UF version on Windows did not have this issue.

Is this expected behavior?

Thanks in advance.

Re: Splunk Universal Forwarder timezone incorrect after daylight saving time change

harsmarvania57 — Fri, 30 Apr 2021 13:46:20 GMT

Hi,

This is not expected behaviour, how are you sending logs to Indexer? From UF -> Indexer OR UF -> Intermediate UF -> Indexer?

Re: Splunk Universal Forwarder timezone incorrect after daylight saving time change

mattbg — Fri, 30 Apr 2021 14:11:40 GMT

UF -> Indexer

It's a small installation; no clustering (yet); all Splunk Enterprise components are on a single node.

Re: Splunk Universal Forwarder timezone incorrect after daylight saving time change

harsmarvania57 — Fri, 30 Apr 2021 14:19:06 GMT

I'll suggest to open support case with splunk.

Re: Splunk Universal Forwarder timezone incorrect after daylight saving time change

mattbg — Thu, 22 Jul 2021 14:46:24 GMT

To close this off, I was advised by support that this is a defect fixed in Splunk 8.1.5 (assume this refers to the UF, but didn't enquire as we have plans to upgrade both Enterprise and the UF to 8.2.x)

Re: Splunk Universal Forwarder timezone incorrect after daylight saving time change

yeahnah — Thu, 18 Aug 2022 22:20:32 GMT

Just experienced this issue on UF 8.1.3 (UF release notes show 8.1.6 has fix) and I'm updating this topic with what we found, in case anyone else comes acrtoss this issue and may find it useful.

DST change occurred (summer time ended, so fallback in our case) on Sunday morning after which we noted this issue. It was only occurring for UF events where the log data did not have timestamped events containing a TZ offset (e.g. +1200). So not all data was affected.

On the Sunday we restarted the UFs/HFs/and even the IDXs to try and resolve the issue, but nothing worked so raised a case with Splunk support.

On Monday, while investigating this issue further, we restarted the UF again and noted that it now fixed the problem and the _time offset was now applied correctly.

So, a UF restarts worked but not until 24 hours after DST change, or maybe until the next day. The UF will still need to be restarted after this time to workaround this bug.

NOTE: this issue occurs for both DST change overs periods - fall back (-1h) and spring forward (+1h).

Re: Splunk Universal Forwarder timezone incorrect after daylight saving time change

mattbg — Mon, 04 Apr 2022 12:30:10 GMT

We saw this again on some of our Windows UFs that had not yet been upgraded past 8.1.2 (it's the 8.1.5 release notes that show the fix).

However, we did not see the issue on any of our UNIX UFs that are on 8.2.5, which is a good sign given that the large majority of our UFs are UNIX.

Re: Why is Splunk Universal Forwarder timezone incorrect after daylight saving time change?

lesliejones3 — Thu, 02 Mar 2023 18:50:07 GMT

I have seen some different versions of the UF mentioned. Is there a specific version and later that resolves the DLST issue. We are coming up on March 12, 2023 and DLST will be back. We have a large number of UF's at 8.2.1 and I'm worried we will see this issue pop up.

Re: Why is Splunk Universal Forwarder timezone incorrect after daylight saving time change?

yeahnah — Thu, 02 Mar 2023 19:53:40 GMT

Hi @lesliejones3

The best place to confirm is via the universal forwarder releases notes (ensure correct major version selected).

https://docs.splunk.com/Documentation/Forwarder/8.2.10/Forwarder/Fixedissues

A quick look under fixed issues shows v8.2.2 has the fix, so, sadly, you will be affected on your current 8.2.1 version.

Hope that helps

Re: Why is Splunk Universal Forwarder timezone incorrect after daylight saving time change?

lesliejones3 — Thu, 02 Mar 2023 19:57:36 GMT

Thank you for the reply. Now it's time to see how many I can upgrade before the 12th. 🙂