Getting Data In

Why is Splunk Universal Forwarder timezone incorrect after daylight saving time change?

mattbg
Path Finder

Using Splunk UF 8.1.1, we've noticed an issue where the Linux x64 forwarder running on RedHat 7.7 did not seem to correctly adjust for daylight saving time; that is, the timestamps after the DST change are 1 hour ahead of where they should be.

We are not using any special TZ configuration on the UF or indexer and have until now relied on the Splunk UF picking up the underlying OS timezone to enrich events which, as I understand from the props.conf spec, is a supported approach.

Simply restarting the UF has resolved the issue on multiple servers.

The same UF version on Windows did not have this issue.

Is this expected behavior?

Thanks in advance.

Labels (3)
0 Karma
1 Solution

mattbg
Path Finder

To close this off, I was advised by support that this is a defect fixed in Splunk 8.1.5 (assume this refers to the UF, but didn't enquire as we have plans to upgrade both Enterprise and the UF to 8.2.x)

View solution in original post

mattbg
Path Finder

To close this off, I was advised by support that this is a defect fixed in Splunk 8.1.5 (assume this refers to the UF, but didn't enquire as we have plans to upgrade both Enterprise and the UF to 8.2.x)

yeahnah
Communicator

Just experienced this issue on UF 8.1.3 (UF release notes show 8.1.6 has fix) and I'm updating this topic with what we found, in case anyone else comes acrtoss this issue and may find it useful.

DST change occurred (summer time ended, so fallback in our case) on Sunday morning after which we noted this issue.  It was only occurring for UF events where the log data did not have timestamped events containing a TZ offset (e.g. +1200).  So not all data was affected. 

On the Sunday we restarted the UFs/HFs/and even the IDXs to try and resolve the issue, but nothing worked so raised a case with Splunk support.

On Monday, while investigating this issue further, we restarted the UF again and noted that it now fixed the problem and the _time offset was now applied correctly.

So, a UF restarts worked but not until 24 hours after DST change, or maybe until the next day.   The UF will still need to be restarted after this time to workaround this bug.



 

0 Karma

mattbg
Path Finder

We saw this again on some of our Windows UFs that had not yet been upgraded past 8.1.2 (it's the 8.1.5 release notes that show the fix).

However, we did not see the issue on any of our UNIX UFs that are on 8.2.5, which is a good sign given that the large majority of our UFs are UNIX.

0 Karma

harsmarvania57
SplunkTrust
SplunkTrust

I'll suggest to open support case with splunk.

0 Karma

harsmarvania57
SplunkTrust
SplunkTrust

Hi,

This is not expected behaviour, how are you sending logs to Indexer? From UF -> Indexer OR UF -> Intermediate UF -> Indexer?

0 Karma

mattbg
Path Finder

UF -> Indexer

It's a small installation; no clustering (yet); all Splunk Enterprise components are on a single node.

0 Karma
Get Updates on the Splunk Community!

NEW! Log Views in Splunk Observability Dashboards Gives Context From a Single Page

Today, Splunk Observability releases log views, a new feature for users to add their logs data from Splunk Log ...

Last Chance to Submit Your Paper For BSides Splunk - Deadline is August 12th!

Hello everyone! Don't wait to submit - The deadline is August 12th! We have truly missed the community so ...

Ready, Set, SOAR: How Utility Apps Can Up Level Your Playbooks!

 WATCH NOW Powering your capabilities has never been so easy with ready-made Splunk® SOAR Utility Apps. Parse ...