Getting Data In

Why is Splunk Universal Forwarder timezone incorrect after daylight saving time change?

mattbg
Path Finder

Using Splunk UF 8.1.1, we've noticed an issue where the Linux x64 forwarder running on RedHat 7.7 did not seem to correctly adjust for daylight saving time; that is, the timestamps after the DST change are 1 hour ahead of where they should be.

We are not using any special TZ configuration on the UF or indexer and have until now relied on the Splunk UF picking up the underlying OS timezone to enrich events which, as I understand from the props.conf spec, is a supported approach.

Simply restarting the UF has resolved the issue on multiple servers.

The same UF version on Windows did not have this issue.

Is this expected behavior?

Thanks in advance.

Labels (3)
0 Karma
1 Solution

mattbg
Path Finder

To close this off, I was advised by support that this is a defect fixed in Splunk 8.1.5 (assume this refers to the UF, but didn't enquire as we have plans to upgrade both Enterprise and the UF to 8.2.x)

View solution in original post

lesliejones3
Loves-to-Learn

I have seen some different versions of the UF mentioned.  Is there a specific version and later that resolves the DLST issue.  We are coming up on March 12, 2023 and DLST will be back.  We have a large number of UF's at 8.2.1 and I'm worried we will see this issue pop up.

0 Karma

yeahnah
Motivator

Hi @lesliejones3 

The best place to confirm is via the universal forwarder releases notes (ensure correct major version selected).

https://docs.splunk.com/Documentation/Forwarder/8.2.10/Forwarder/Fixedissues

A quick look under fixed issues shows v8.2.2 has the fix, so, sadly, you will be affected on your current 8.2.1 version.

yeahnah_0-1677786666816.png 

Hope that helps

Tags (1)
0 Karma

lesliejones3
Loves-to-Learn

Thank you for the reply.  Now it's time to see how many I can upgrade before the 12th.  🙂

 

0 Karma

mattbg
Path Finder

To close this off, I was advised by support that this is a defect fixed in Splunk 8.1.5 (assume this refers to the UF, but didn't enquire as we have plans to upgrade both Enterprise and the UF to 8.2.x)

yeahnah
Motivator

Just experienced this issue on UF 8.1.3 (UF release notes show 8.1.6 has fix) and I'm updating this topic with what we found, in case anyone else comes acrtoss this issue and may find it useful.

DST change occurred (summer time ended, so fallback in our case) on Sunday morning after which we noted this issue.  It was only occurring for UF events where the log data did not have timestamped events containing a TZ offset (e.g. +1200).  So not all data was affected. 

On the Sunday we restarted the UFs/HFs/and even the IDXs to try and resolve the issue, but nothing worked so raised a case with Splunk support.

On Monday, while investigating this issue further, we restarted the UF again and noted that it now fixed the problem and the _time offset was now applied correctly.

So, a UF restarts worked but not until 24 hours after DST change, or maybe until the next day.   The UF will still need to be restarted after this time to workaround this bug.

NOTE: this issue occurs for both DST change overs periods - fall back (-1h) and spring forward (+1h).


0 Karma

mattbg
Path Finder

We saw this again on some of our Windows UFs that had not yet been upgraded past 8.1.2 (it's the 8.1.5 release notes that show the fix).

However, we did not see the issue on any of our UNIX UFs that are on 8.2.5, which is a good sign given that the large majority of our UFs are UNIX.

0 Karma

harsmarvania57
SplunkTrust
SplunkTrust

I'll suggest to open support case with splunk.

0 Karma

harsmarvania57
SplunkTrust
SplunkTrust

Hi,

This is not expected behaviour, how are you sending logs to Indexer? From UF -> Indexer OR UF -> Intermediate UF -> Indexer?

0 Karma

mattbg
Path Finder

UF -> Indexer

It's a small installation; no clustering (yet); all Splunk Enterprise components are on a single node.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...