topic Re: Why is my TIME_FORMAT regular expression in props.conf not working for an index time extraction? in Splunk Search

Why is my TIME_FORMAT regular expression in props.conf not working for an index time extraction?

sreejith2k2 — Fri, 17 Mar 2017 12:46:59 GMT

HI I am using following regular expression for the index time extraction in the props.conf. For some reason, it is not extracting properly.

Event: 2017-03-15T11:30:02.609835+00:00 postfix/pickup[19819]: 89389386: uid=0 from user1

I have defined my sourcetype as mail.

[source::...mail]
sourcetype=mail

[mail]
SHOULD_LINEMERGE = false
TIME_FORMAT = %Y-%m-%dT%H:%M:%S.%6Q%:z
TIME_PREFIX = ^
LINE_BREAKER = ([\r\n]+)
MAX_TIMESTAMP_LOOKAHEAD = 30
TRUNCATE=5000

Also, is there any websites i can test the TIME_FORMAT regular expression similar to regex101.com?

Re: Why is my TIME_FORMAT regular expression in props.conf not working for an index time extraction?

richgalloway — Fri, 17 Mar 2017 13:08:20 GMT

The TIME_FORMAT attribute does not use regex strings.
I'm not aware of any website for testing time format strings, but you can do it in a search window. Try something like this:

| makeresults | eval TS="2017-03-15T11:30:02.609835+00:00" | eval epoch=strptime(TS,"%Y-%m-%dT%H:%M:%S.%6Q%:z") | table TS epoch

If the format string is bad, epoch will be null.

Your format string works for me in search, but I haven't tried it at index time. You might want to try this alternative: "%Y-%m-%dT%H:%M:%S.%6N%z".

Re: Why is my TIME_FORMAT regular expression in props.conf not working for an index time extraction?

jwelch_splunk — Tue, 29 Sep 2020 13:18:34 GMT

What is the source of the data? is it /var/log/mail/*.log and how is it coming to the indexer? Is there a UF

Other things I noticed:
increase MAX_TIMESTAMP_LOOKAHEAD = 32

And it is always best to do your LINE_BREAKER like so
LINE_BREAKER = ([\r\n]+)\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}.\d{6}+\d{2}:\d{2}

Re: Why is my TIME_FORMAT regular expression in props.conf not working for an index time extraction?

sreejith2k2 — Tue, 29 Sep 2020 13:18:32 GMT

HI Welch,

Thanks for the answer. There were 2 issues

MAX_TIMESTAMP_LOOKAHEAD
[source::...mail] - i have put only 2 dots in my props.conf instead of 3.

Re: Why is my TIME_FORMAT regular expression in props.conf not working for an index time extraction?

sreejith2k2 — Mon, 20 Mar 2017 09:35:44 GMT

Thanks Rich for your search. This search helped me in reducing the time in doing the testing..