topic Re: Manually configure timestamp at index time from custom datetime.xml in Getting Data In

Manually configure timestamp at index time from custom datetime.xml

bizza — Mon, 05 May 2014 09:18:30 GMT

I tried to configure a custom datetime.xml (for my first time) as follow:

<datetime>

<define name="csdate" extract="year, month, day, hour, minute">
        <text><![CDATA[[\s\S]{40}(\d{4})(\d{2})(\d{2})[\s\S]{206}(\d{2})(\d{2})]]></text> 
</define>

<timePatterns>
    <use name="csdate"/>
</timePatterns> 

<datePatterns>
    <use name="csdate"/>
</datePatterns>
</datetime>

Regex match exactly year, mont, day, hour and minute.
In props.conf I added

DATETIME_CONFIG = /etc/system/local/datetime.xml

SHOULD_LINEMERGE = FALSE

TIME_FORMAT = %Y%m%d%H%M

Any ideas why data are not indexed with resulting timestamp?
I tried to split in 2 regex, for timePatterns and datePatterns match, but the result is still the same.

Or do you suggest a different way to achieve timestamp override at index time?

Regards

Re: Manually configure timestamp at index time from custom datetime.xml

lguinn2 — Mon, 05 May 2014 19:09:57 GMT

You don't need a custom datetime.xml - I wouldn't do it that way. It is complicated and unnecessary.

In props.conf all you should need is

SHOULD_LINEMERGE = FALSE
TIME_FORMAT = %Y%m%d%H%M

Assuming that your timestamp looks like

201405021209

If not, please comment with an example or two of the timestamp.

Re: Manually configure timestamp at index time from custom datetime.xml

bizza — Mon, 05 May 2014 19:16:59 GMT

The problem is that timestamp is splitted on every lines.
For example:

204023600511105443000 20140422--Y2014--m04--d180000000005.0600000000000041096125031ABDCE 81234567 ABDCE F & C 10024 ABDCE F & C 45399700123456789000000000.104023600582105443000 386511186636492--H15--M36PSBP

every line has 300 characters(digits), fields are position-sensitive.
I added --Y, --m, --d, --H and --M just before timestamp fields.

I believe that a custom datetime.xml is my only option.

Re: Manually configure timestamp at index time from custom datetime.xml

lmyrefelt — Mon, 28 Sep 2020 16:32:23 GMT

Did you add the --Y and --m into the event example as an clarification ?
otherwise you could try;
TIME_PREFIX = \d{numberOfDigits}\s++
MAX_TIMESTAMP_LOOKAHEAD = 20
TIME_FORMAT = %Y%m%d
TIME_FORMAT = --Y%Y--m%m--d%d

Re: Manually configure timestamp at index time from custom datetime.xml

bizza — Mon, 05 May 2014 20:48:33 GMT

Yes, I added the --Y --m ecc only to show where timestamp fields are.
Ignore it and you'll have the original log line.

Re: Manually configure timestamp at index time from custom datetime.xml

abonuccelli_spl — Mon, 28 Sep 2020 16:44:43 GMT

Problem here I believe was due to actual timestamps in raw event past the default (MAX_TIMESTAMP_LOOKAHEAD) 150 chars.

Re: Manually configure timestamp at index time from custom datetime.xml

marcoscala — Wed, 04 Jun 2014 08:22:02 GMT

Antonio,
I'm afraid but that's not the case either. In his/her case, date and time are splitted in the event data, so usual timeformat is more complex to manage. Unfortunately we can't use REGEX for TIME_FORMAT, otherwise that was the solution.

Marco

Re: Manually configure timestamp at index time from custom datetime.xml

abonuccelli_spl — Wed, 04 Jun 2014 08:32:55 GMT

MMM, well it works for me...
Bizza should be able to confirm
Antonio

Re: Manually configure timestamp at index time from custom datetime.xml

bizza — Mon, 28 Sep 2020 16:47:53 GMT

Yes, problem was MAX_TIMESTAMP_LOOKAHEAD.
Thanks for your help guys

ciao

Re: Manually configure timestamp at index time from custom datetime.xml

nitesh218ss — Tue, 12 May 2015 06:09:33 GMT

Hi in my log event and filename date is not present i want give a fix date to log so what is do ?