The timestamp has a one hour offset

[djoiret]

Hello,

I am using "Splunk_TA_juniper" and I noticed a new problem with timestamp: there is a one hour offset for the timestamp compared to the time in the event. For instance, when I have an event whose _raw value starts with "Oct 28 15:12:37 fw-01-gra RT_FLOW:  ...", the timestamps is "2021-10-28T16:12:37.000+02:00" (16h instead of 15h). In addition, the event will only appear after an hour after its received by the indexer, in fact when the timestamp value is less than the current time.

This behaviour is new. When I examine  events for september (for instance), the timestamp matches the time in the event.

I tried to restart Splunk and the forwarder, nothing was changed. I haven't modify the configuration files for a long time, and I don't know what to do.

Do you have an idea of what is going on or a possible solution?

Regards

Denis

[djoiret]

Hello,

The problem is not from the sources. I have several logs sources (Juniper logs, "authpriv" from Linux servers, Cisco ESA logs, etc.) and all had this 1 hour offset between 10/28 and 10/31 02:00:00. You can see in the joined image two events, one just before 02:00:00 (at 01:59:59) and the second one at 02:00:00. The timestamp for the first event is wrong, the timestamp for the second event is correct.

Denis

[richgalloway]

It sounds like the country where the source resides recently changed from Summer time to standard time, but the source is still reporting timestamps in Summer time.  It could be the application at fault or a Splunk setting.  Can you share the props.conf settings for that sourcetype?

[djoiret]

Hello,

After some other investigations, I noticed that in fact ALL events having  syslog format for date and time had this 1 hour offset. The problem started on 10/28/2021 00:00:00 until 10/30/2021 23:59:59. All events after  10/31/2021 00:00:00 have a correct timestamp, so the problem just disappeared.

In France we changed from summer time to winter time on Sun 10/31/2021 at 03:00:00 (at 03:00:00, local time was changed to 02:00:00).

I never noticed this problem before, I have been using Splunk since 2014. In my opinion, this looks like a bug in Splunk. I know no algorithm for changing summer time to winter time taking place on tuesday.

For the application Splunk_TA_juniper, there was no parameters about time in "default/props.conf". There are no parameter either in system props.conf.

I added these parameters in "local/props.conf" of application Splunk_TA_juniper before my previous post :

[juniper:junos:firewall]
TIME_FORMAT = "%b %e %T"
TZ = Europe/Paris

This did not change anything.

Denis

[isoutamo]

Hi
I'm not sure if this related anything in your case, but we have noticed that there are still some network equipments, which cannot handle timestamps correctly with syslog feed when summertime starts or ends. Usually it has require reboot and time by time even this haven't helps. There could be two different timestamp in event and one is right and second one is not.
r. Ismo