Why is my TIME_FORMAT regular expression in props.conf not working for an index time extraction?

[sreejith2k2]

HI I am using following regular expression for the index time extraction in the props.conf. For some reason, it is not extracting properly.

Event: 2017-03-15T11:30:02.609835+00:00 postfix/pickup[19819]: 89389386: uid=0 from user1

I have defined my sourcetype as mail.

[source::...mail]
sourcetype=mail

[mail]
SHOULD_LINEMERGE = false
TIME_FORMAT = %Y-%m-%dT%H:%M:%S.%6Q%:z
TIME_PREFIX = ^
LINE_BREAKER = ([\r\n]+)
MAX_TIMESTAMP_LOOKAHEAD = 30
TRUNCATE=5000

Also, is there any websites i can test the TIME_FORMAT regular expression similar to regex101.com?

[jwelch_splunk · Accepted solution]

What is the source of the data? is it /var/log/mail/*.log and how is it coming to the indexer? Is there a UF

Other things I noticed:
increase MAX_TIMESTAMP_LOOKAHEAD = 32

And it is always best to do your LINE_BREAKER like so
LINE_BREAKER = ([\r\n]+)\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}.\d{6}+\d{2}:\d{2}

[sreejith2k2]

HI Welch,

Thanks for the answer. There were 2 issues

1. MAX_TIMESTAMP_LOOKAHEAD
2. [source::...mail] - i have put only 2 dots in my props.conf instead of 3.

[richgalloway]

The TIME_FORMAT attribute does not use regex strings.
I'm not aware of any website for testing time format strings, but you can do it in a search window. Try something like this:

| makeresults | eval TS="2017-03-15T11:30:02.609835+00:00" | eval epoch=strptime(TS,"%Y-%m-%dT%H:%M:%S.%6Q%:z") | table TS epoch

If the format string is bad, epoch will be null.

Your format string works for me in search, but I haven't tried it at index time. You might want to try this alternative: "%Y-%m-%dT%H:%M:%S.%6N%z".

[sreejith2k2]

Thanks Rich for your search. This search helped me in reducing the time in doing the testing..