Why is some data source indexed one hour in the future?

emallinger — Fri, 15 Apr 2022 17:35:25 GMT

Hi,

Some data source is indexed one hour in the future (probably since TZ shift => twice a year hour change in France !! this time +0100hour).

We were on gmt+1, now we're on gmt+2.

I don't know where the problem is.
- checked the server ntp => ok, gmt+2 updated
- checked the data source file => ok
- tried to reproduced in dev env on a mono-instance : issue not reproducted !

- this is the only data source with the issue

My prod env is distributed (SHC, Indexer Cluster and multiple forwarders)

- data is a jsonl file.

I'm soo lost !!

Thank you for your help !!

Ema

on the indexer cluster :

[mysourcetype]
NO_BINARY_CHECK = true
SHOULD_LINEMERGE = false
TIME_PREFIX= "dte":"
TIME_FORMAT = %d/%m/%Y %H:%M:%S
TRUNCATE = 0
MAX_DAYS_AGO = 4000
category = Structured
disabled = false
pulldown_type = true

data sample :

{"idj":"3108824152","dce":"IDN","fce":"IDN2","ace":"176","dte":"08/04/2022 14:44:31","org":"GN","dmc":"2","idu":"211151","csu":"00082827","lsu":"CROSS BDOHRIJ GHBGD14 ","ctx":"Identifiant:PN-003042021007790-ARD-PPM-70732201#Procédure de référence:CIAHTDT CENTRAL DE CNJAEN-2021-007790#Type personne:Physique#Qualité personne:Mise en cause#Nom:XXX#Prénom:yyy#Lieu de naissance:CAEN#Date de naissance:05/01/1991#","idd":"PN-0030428541021007790-ARD-PPM-7074532201","ise":"N","cts":[{"idj":"3108824152","nom":"XXX","pre":"yyy","jne":"5","mne":"1","ane":"1981","lne":"CAEN","cot":"","not":"","qot":"","nuo":"","ctt":"","gtt":"","qtt":"","ntt":""}]}

This data is indexed at 08/04/2022 15:44:31 for 08/04/2022 14:44:31 !

Re: Some data source is indexed one hour in the future

PickleRick — Fri, 15 Apr 2022 15:46:27 GMT

Your source doesn't report the timezone so the indexer/hf, since no tz is defined for the sourcetype, interprets with its local timezone.

Re: Some data source is indexed one hour in the future

emallinger — Tue, 19 Apr 2022 08:27:13 GMT

Hello,

Yes, but source and indexer are in the same time zone, at the same hour.

So why the difference ?

Thanks for your suggestions,

Ema

topic Why is some data source indexed one hour in the future? in Getting Data In

Why is some data source indexed one hour in the future?

Re: Some data source is indexed one hour in the future

Re: Some data source is indexed one hour in the future