Getting Data In

Why is Splunk Universal Forwarder timezone incorrect after daylight saving time change?

mattbg
Path Finder

Using Splunk UF 8.1.1, we've noticed an issue where the Linux x64 forwarder running on RedHat 7.7 did not seem to correctly adjust for daylight saving time; that is, the timestamps after the DST change are 1 hour ahead of where they should be.

We are not using any special TZ configuration on the UF or indexer and have until now relied on the Splunk UF picking up the underlying OS timezone to enrich events which, as I understand from the props.conf spec, is a supported approach.

Simply restarting the UF has resolved the issue on multiple servers.

The same UF version on Windows did not have this issue.

Is this expected behavior?

Thanks in advance.

Labels (3)
0 Karma
1 Solution

mattbg
Path Finder

To close this off, I was advised by support that this is a defect fixed in Splunk 8.1.5 (assume this refers to the UF, but didn't enquire as we have plans to upgrade both Enterprise and the UF to 8.2.x)

View solution in original post

lesliejones3
Loves-to-Learn

I have seen some different versions of the UF mentioned.  Is there a specific version and later that resolves the DLST issue.  We are coming up on March 12, 2023 and DLST will be back.  We have a large number of UF's at 8.2.1 and I'm worried we will see this issue pop up.

0 Karma

yeahnah
Motivator

Hi @lesliejones3 

The best place to confirm is via the universal forwarder releases notes (ensure correct major version selected).

https://docs.splunk.com/Documentation/Forwarder/8.2.10/Forwarder/Fixedissues

A quick look under fixed issues shows v8.2.2 has the fix, so, sadly, you will be affected on your current 8.2.1 version.

yeahnah_0-1677786666816.png 

Hope that helps

Tags (1)
0 Karma

lesliejones3
Loves-to-Learn

Thank you for the reply.  Now it's time to see how many I can upgrade before the 12th.  🙂

 

0 Karma

mattbg
Path Finder

To close this off, I was advised by support that this is a defect fixed in Splunk 8.1.5 (assume this refers to the UF, but didn't enquire as we have plans to upgrade both Enterprise and the UF to 8.2.x)

yeahnah
Motivator

Just experienced this issue on UF 8.1.3 (UF release notes show 8.1.6 has fix) and I'm updating this topic with what we found, in case anyone else comes acrtoss this issue and may find it useful.

DST change occurred (summer time ended, so fallback in our case) on Sunday morning after which we noted this issue.  It was only occurring for UF events where the log data did not have timestamped events containing a TZ offset (e.g. +1200).  So not all data was affected. 

On the Sunday we restarted the UFs/HFs/and even the IDXs to try and resolve the issue, but nothing worked so raised a case with Splunk support.

On Monday, while investigating this issue further, we restarted the UF again and noted that it now fixed the problem and the _time offset was now applied correctly.

So, a UF restarts worked but not until 24 hours after DST change, or maybe until the next day.   The UF will still need to be restarted after this time to workaround this bug.

NOTE: this issue occurs for both DST change overs periods - fall back (-1h) and spring forward (+1h).


0 Karma

mattbg
Path Finder

We saw this again on some of our Windows UFs that had not yet been upgraded past 8.1.2 (it's the 8.1.5 release notes that show the fix).

However, we did not see the issue on any of our UNIX UFs that are on 8.2.5, which is a good sign given that the large majority of our UFs are UNIX.

0 Karma

harsmarvania57
Ultra Champion

I'll suggest to open support case with splunk.

0 Karma

harsmarvania57
Ultra Champion

Hi,

This is not expected behaviour, how are you sending logs to Indexer? From UF -> Indexer OR UF -> Intermediate UF -> Indexer?

0 Karma

mattbg
Path Finder

UF -> Indexer

It's a small installation; no clustering (yet); all Splunk Enterprise components are on a single node.

0 Karma
Get Updates on the Splunk Community!

2024 Splunk Career Impact Survey | Earn a $20 gift card for participating!

Hear ye, hear ye! The time has come again for Splunk's annual Career Impact Survey!  We need your help by ...

Optimize Cloud Monitoring

  TECH TALKS Optimize Cloud Monitoring Tuesday, August 13, 2024  |  11:00AM–12:00PM PST   Register to ...

What's New in Splunk Cloud Platform 9.2.2403?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.2.2403! Analysts can ...