Getting Data In
Highlighted

Manually configure timestamp at index time from custom datetime.xml

Path Finder

I tried to configure a custom datetime.xml (for my first time) as follow:

<datetime>

<define name="csdate" extract="year, month, day, hour, minute">
        <text><![CDATA[[\s\S]{40}(\d{4})(\d{2})(\d{2})[\s\S]{206}(\d{2})(\d{2})]]></text> 
</define>

<timePatterns>
    <use name="csdate"/>
</timePatterns> 

<datePatterns>
    <use name="csdate"/>
</datePatterns>
</datetime>

Regex match exactly year, mont, day, hour and minute.
In props.conf I added

DATETIME_CONFIG = /etc/system/local/datetime.xml

SHOULD_LINEMERGE = FALSE

TIME_FORMAT = %Y%m%d%H%M

Any ideas why data are not indexed with resulting timestamp?
I tried to split in 2 regex, for timePatterns and datePatterns match, but the result is still the same.

Or do you suggest a different way to achieve timestamp override at index time?

Regards

Highlighted

Re: Manually configure timestamp at index time from custom datetime.xml

Legend

You don't need a custom datetime.xml - I wouldn't do it that way. It is complicated and unnecessary.

In props.conf all you should need is

SHOULD_LINEMERGE = FALSE
TIME_FORMAT = %Y%m%d%H%M

Assuming that your timestamp looks like

201405021209

If not, please comment with an example or two of the timestamp.

0 Karma
Highlighted

Re: Manually configure timestamp at index time from custom datetime.xml

Path Finder

The problem is that timestamp is splitted on every lines.
For example:

204023600511105443000 20140422--Y2014--m04--d180000000005.0600000000000041096125031ABDCE 81234567 ABDCE F & C 10024 ABDCE F & C 45399700123456789000000000.104023600582105443000 386511186636492--H15--M36PSBP

every line has 300 characters(digits), fields are position-sensitive.
I added --Y, --m, --d, --H and --M just before timestamp fields.

I believe that a custom datetime.xml is my only option.

0 Karma
Highlighted

Re: Manually configure timestamp at index time from custom datetime.xml

Builder

Did you add the --Y and --m into the event example as an clarification ?
otherwise you could try;
TIMEPREFIX = \d{numberOfDigits}\s++
MAX
TIMESTAMPLOOKAHEAD = 20
TIME
FORMAT = %Y%m%d
TIME_FORMAT = --Y%Y--m%m--d%d

0 Karma
Highlighted

Re: Manually configure timestamp at index time from custom datetime.xml

Path Finder

Yes, I added the --Y --m ecc only to show where timestamp fields are.
Ignore it and you'll have the original log line.

0 Karma
Highlighted

Re: Manually configure timestamp at index time from custom datetime.xml

Communicator

Hi in my log event and filename date is not present i want give a fix date to log so what is do ?

0 Karma
Highlighted

Re: Manually configure timestamp at index time from custom datetime.xml

Splunk Employee
Splunk Employee

Problem here I believe was due to actual timestamps in raw event past the default (MAXTIMESTAMPLOOKAHEAD) 150 chars.

View solution in original post

Highlighted

Re: Manually configure timestamp at index time from custom datetime.xml

Builder

Antonio,
I'm afraid but that's not the case either. In his/her case, date and time are splitted in the event data, so usual timeformat is more complex to manage. Unfortunately we can't use REGEX for TIME_FORMAT, otherwise that was the solution.

Marco

0 Karma
Highlighted

Re: Manually configure timestamp at index time from custom datetime.xml

Splunk Employee
Splunk Employee

MMM, well it works for me...
Bizza should be able to confirm
Antonio

0 Karma
Highlighted

Re: Manually configure timestamp at index time from custom datetime.xml

Path Finder

Yes, problem was MAXTIMESTAMPLOOKAHEAD.
Thanks for your help guys

ciao

0 Karma