Why are events indexing with the wrong time stamp?

[freddy_Guo]

Hi,
We have a custom TA to collect some logs from a Windows Server.
This morning I just noticed that the Splunk is actually swapping day and month.
Note: The time difference is from different time zone, shouldn't be a problem
For example

1/11/22 9:59:30.447 AM Src1 [01/11/2022 08:59:30.447]
1/11/22 9:59:30.447 AM Src1 [01/11/2022 08:59:30.447]

It was working before the event time turned to 01/11/2022 00:00:00
Last logging:

11/1/22 12:59:30.548 AM Src1 [31/10/2022 23:59:30.548]

Our props.conf looks like below:

DATETIME_CONFIG =
LINE_BREAKER = ([\r\n]+)
TIME_FORMAT = %d/%m/%Y %H:%M:%S
MAX_TIMESTAMP_LOOKAHEAD = 125
SHOULD_LINEMERGE = false
TIME_PREFIX = Src1\s+\[

 Any suggestion will be appreciated. Thanks

[gcusello]

Hi @freddy_Guo,

where do you located this TA?

it must be located on Indexers or, if present, on Heavy Forwarders, not on Universal Forwarders.

Ciao.

Giuseppe

[freddy_Guo]

Hi @gcusello ,

Thanks for the reply. Good question. It's being deployed on the Universal Forwarder by Deployment server and I don't think it's on indexers. 

But it was working just fine before, just don't understand how that date change would affect this. 

Cheers,

Freddy

[gcusello · Accepted solution]

Hi @freddy_Guo,

the problem is that you have a date in european format (dd/mm/yyyy) instead Splunk by default works in american format (mm/dd/yyyy) so until the 31 of october it correctly interpretes the date format, but from the 1st until the 11th of the month it uses the wrong format.

So you need to give the correct TIME_FORMAT on the Indexers or (if present) on Heavy Forwarders.

Ciao.

Giuseppe

[freddy_Guo]

Hi @gcusello 

Again, I really apprecaite your help. The logging is back now after I pushed the TA contains that props.conf into our indexer cluster, you were right. Indexers had no idea what time the should use.

But one thing that still doesn't make sense to me is why it doesn't know how to handle logs from the 1st until the 11th of the month

[gcusello]

Hi @freddy_Guo,

as I said, by default Splunk uses the american date format (mm/dd/yyyy) so from the 1st to the 12th of the month it can apply this date format to your events, after the 12th it recognizes that the american format is wrong and it applies the european format.

The solution is to say to Splunk that for that sourcetype it has to apply the european format using TIME_FORMAT option in props.conf.

Tell me if I can help you more, otherwise, please, accept one answer for the other people of Community.

Ciao and happy splunking.

Giuseppe

P.S.: Karma Points are appreciated 😉

[freddy_Guo]

Hi @gcusello ,

Thank you for the detailed explanation!

Done done. 

Cheers,

Freddy

[gcusello]

Hi @freddy_Guo,

good for you, see next time!

Ciao and happy splunking

Giuseppe

P.S.: Karma Points are appreciated 😉

[freddy_Guo]

Hi @gcusello  

Thank you so much for the answer.  That's very interesting. I thought having the TA in universal forwarder would make sure the date extracted properly. 

Thank you. I will deploy this TA on our indexer clusters. But does the props.conf look good to you?