topic Re: [Extract DateTime in event to _time] datetime.xml some formats in Getting Data In

[Extract DateTime in event to _time] datetime.xml some formats

SuperMisterT — Tue, 24 Aug 2021 13:40:11 GMT

Hi,

I have TCP 514 logs in the same sourcetype.
There are different formats of timestamp in log and even in events.
I don't understand my mistakes with datetime.xml. It's working for one format but not for the second.

I text regexp with search ( | rex field=_raw ".........") fields are correctly extracted.

I follow thus tuto: https://www.function1.com/2013/01/oh-no-splunking-log-files-with-multiple-formats-no-problem

Thanks for your help.

Example:
first log:
<111> YYYY-MM-DDTHH:MM:SS+02:00 localhost house 12154 - @ip [DD/LitMM/YYYY:HH:MM:SS.MS] ...........
_time is correctly extract,

second log:
<145> YYYY-MM-DDTHH:MM:SS+02:00 localhost foo - - YYYY-MM-DDTHH:MM:SS.MS+0000 jizjfoziejfz battle: cececeijoijoi [YYYY-MM-DDTHH:MM:SS.MS+0000] ...........
_time is not extracted, value is index time 😞

I'm on a standalone station, so i copy regexp without storage (maybe typo).

Configuration:
in datetime.xml on HeayFW (etc/apps/test/default)

<define name="_house" extract="day, litmonth,year,hour,minute,second,subsecond">
<text>house.*\[(\d{2})/(\w{3})/(\d{4}):(\d{2}):(\d{2}):(\d{2})\.\d+\]></text>
</define>
<define name="_battle" extract="year,month,day,hour,minute,second,subsecond">
<text>battle.*\[(\d{4})\-(\d{2})-(\d{2})T(\d{2}):(\d{2}):(\d{2})\.\d+\+\d{4}\]></text>
</define>

in props.conf
[my_sourcetype]
DATETIME_CONGIG= /etc/apps/test/defaults/datetime.xml
LINE_BREAKER = ([\r|\n])+
SHOULD_LINEMERGE = false

Re: [Extract DateTime in event to _time] datetime.xml some formats

harsmarvania57 — Tue, 24 Aug 2021 14:15:57 GMT

I have not gone through regex in details but I can see that in your datetime.xml, you don't have <datetime> and in props.conf you have defined defaults however you have directory called default

Re: [Extract DateTime in event to _time] datetime.xml some formats

SuperMisterT — Tue, 24 Aug 2021 14:26:18 GMT

Yes this is a typo when i copied my screen sorry.....

Re: [Extract DateTime in event to _time] datetime.xml some formats

harsmarvania57 — Tue, 24 Aug 2021 14:53:46 GMT

Can you please try below config in datetime.xml?

Re: [Extract DateTime in event to _time] datetime.xml some formats

SuperMisterT — Tue, 24 Aug 2021 14:42:33 GMT

I found the error. This is in the configuration of MAX_TIMESTAMP_LOOKAHEAD.

My field is beyong 128 char. So in log, i see Failed to parse timestamp in first MTL (128).

My question : "how configure tow differents MAX_TIMESTAMP_LOOKAHEAD?"

Re: [Extract DateTime in event to _time] datetime.xml some formats

harsmarvania57 — Tue, 24 Aug 2021 14:51:29 GMT

You can't configure different MAX_TIMESTAMP_LOOKAHEAD. Have you tried datetime.xml which I have provided?