topic timeformat are not getting extracted properly in Getting Data In

timeformat are not getting extracted properly

ashikuma — Tue, 29 Sep 2020 20:54:07 GMT

timeformat are not getting extracted properly, we have one type of timestamp but clock there is different. It is starting from 0 - 24 hours and date starting from 1- 31, and also same for month 1- 12, see timestamp example below.

[8/10/18 0:20:37:469 EDT]
[8/9/18 11:59:59:796 EDT]
[8/9/18 13:16:38:194 EDT]
[8/12/18 1:49:08:943 EDT]
[8/11/18 22:59:45:370 EDT]

I tried to use this props.conf but didn't work

[sourcetypename]
BREAK_ONLY_BEFORE = [\d+\/\d+\/\d+\s\d+[:]\d+[:]\d+[:]\d+\s\w{3}]
TIME_FORMAT = %m/%e/%y %k:%M:%S:%3N

After this I tried to extract using datetime.xml, that is working for some extent but not fully. Using that I am getting delay in indexed event timestamp, please help...

Re: timeformat are not getting extracted properly

cpetterborg — Sun, 12 Aug 2018 16:03:37 GMT

Not sure why you are using %e instead of %d, nor why you aren't using %Z. I doubt that is what is causing your problem, however.

Do you have someplace in your props.conf files a directive to use the index time as the timestamp, which could be superseding the above configuration? Look for that sourcetype or other indication of a configuration clause that could be overriding things. You probably should look at using btool. Something like:

splunk btool props list --debug

Then look at the output for the sourcetype and see if it is using the configuration you expect it to be using.

Re: timeformat are not getting extracted properly

acharlieh — Mon, 13 Aug 2018 02:24:18 GMT

So I decided to play with taking those timestamps and uploading it to my local instance through the Add Data page

Something that was interesting to me, with these dates on my laptop's 7.1.2 instance, it seems like Splunk was getting tripped up by the leading square bracket and or single digit date... which I was able to fix in two different ways... either by specifying a TIME_PREFIX = \[ or starting your TIME_FORMAT with [. If you specify TIME_PREFIX, you could also reduce the search space Splunk uses to identify a timestamp by setting MAX_TIMESTAMP_LOOKAHEAD down to around 26 or 28. If the timestamp is the first thing in your event, then you could even anchor it with TIME_PREFIX = ^\[

As @cpetterborg mentions since your data has timezones, you really should be including %Z in your format... and I would also encourage use of the more familiar %d and %Hover the %e and %k equivalences (but the bracket fix it seems to work either way).

Additionally, if you can guarantee only single line events, I'd encourage making use of SHOULD_LINEMERGE = false instead of BREAK_ONLY_BEFORE = regex or the default BREAK_ONLY_BEFORE_DATE = true. As you have what seems to be an event-breaking regex, you may even consider altering LINE_BREAKER to have only single "line" events, but that's assuming that this timestamp is the first thing after a linebreak in your file. (if you go that route you might also need to be aware that TRUNCATE has a maximum default 10000 bytes per "line"