topic Re: Events indexing with the wrong time stamp in Getting Data In

Why are events indexing with the wrong time stamp?

freddy_Guo — Tue, 01 Nov 2022 14:41:37 GMT

Hi,
We have a custom TA to collect some logs from a Windows Server.
This morning I just noticed that the Splunk is actually swapping day and month.
Note: The time difference is from different time zone, shouldn't be a problem
For example

1/11/22 9:59:30.447 AM Src1 [01/11/2022 08:59:30.447] 1/11/22 9:59:30.447 AM Src1 [01/11/2022 08:59:30.447]

It was working before the event time turned to 01/11/2022 00:00:00
Last logging:

11/1/22 12:59:30.548 AM Src1 [31/10/2022 23:59:30.548]

Our props.conf looks like below:

DATETIME_CONFIG = LINE_BREAKER = ([\r\n]+) TIME_FORMAT = %d/%m/%Y %H:%M:%S MAX_TIMESTAMP_LOOKAHEAD = 125 SHOULD_LINEMERGE = false TIME_PREFIX = Src1\s+\[

Any suggestion will be appreciated. Thanks

Re: Events indexing with the wrong time stamp

gcusello — Tue, 01 Nov 2022 06:53:32 GMT

Hi @freddy_Guo,

where do you located this TA?

it must be located on Indexers or, if present, on Heavy Forwarders, not on Universal Forwarders.

Ciao.

Giuseppe

Re: Events indexing with the wrong time stamp

freddy_Guo — Tue, 01 Nov 2022 22:37:44 GMT

Hi @gcusello ,

Thanks for the reply. Good question. It's being deployed on the Universal Forwarder by Deployment server and I don't think it's on indexers.

But it was working just fine before, just don't understand how that date change would affect this.

Cheers,

Freddy

Re: Events indexing with the wrong time stamp

gcusello — Wed, 02 Nov 2022 07:35:32 GMT

Hi @freddy_Guo,

the problem is that you have a date in european format (dd/mm/yyyy) instead Splunk by default works in american format (mm/dd/yyyy) so until the 31 of october it correctly interpretes the date format, but from the 1st until the 11th of the month it uses the wrong format.

So you need to give the correct TIME_FORMAT on the Indexers or (if present) on Heavy Forwarders.

Ciao.

Giuseppe

Re: Events indexing with the wrong time stamp

freddy_Guo — Thu, 03 Nov 2022 00:18:29 GMT

Hi @gcusello

Thank you so much for the answer. That's very interesting. I thought having the TA in universal forwarder would make sure the date extracted properly.

Thank you. I will deploy this TA on our indexer clusters. But does the props.conf look good to you?

Re: Events indexing with the wrong time stamp

freddy_Guo — Thu, 03 Nov 2022 02:17:44 GMT

Hi @gcusello

Again, I really apprecaite your help. The logging is back now after I pushed the TA contains that props.conf into our indexer cluster, you were right. Indexers had no idea what time the should use.

But one thing that still doesn't make sense to me is why it doesn't know how to handle logs from the 1st until the 11th of the month

Re: Events indexing with the wrong time stamp

gcusello — Thu, 03 Nov 2022 07:11:11 GMT

Hi @freddy_Guo,

as I said, by default Splunk uses the american date format (mm/dd/yyyy) so from the 1st to the 12th of the month it can apply this date format to your events, after the 12th it recognizes that the american format is wrong and it applies the european format.

The solution is to say to Splunk that for that sourcetype it has to apply the european format using TIME_FORMAT option in props.conf.

Tell me if I can help you more, otherwise, please, accept one answer for the other people of Community.

Ciao and happy splunking.

Giuseppe

P.S.: Karma Points are appreciated 😉

Re: Events indexing with the wrong time stamp

freddy_Guo — Thu, 03 Nov 2022 21:10:37 GMT

Hi @gcusello ,

Thank you for the detailed explanation!

Done done.

Cheers,

Freddy

Re: Events indexing with the wrong time stamp

gcusello — Fri, 04 Nov 2022 07:28:45 GMT

Hi @freddy_Guo,

good for you, see next time!

Ciao and happy splunking

Giuseppe

P.S.: Karma Points are appreciated 😉