Getting Data In

Manually configure timestamp at index time from custom datetime.xml

bizza
Path Finder

I tried to configure a custom datetime.xml (for my first time) as follow:

<datetime>

<define name="csdate" extract="year, month, day, hour, minute">
        <text><![CDATA[[\s\S]{40}(\d{4})(\d{2})(\d{2})[\s\S]{206}(\d{2})(\d{2})]]></text> 
</define>

<timePatterns>
    <use name="csdate"/>
</timePatterns> 

<datePatterns>
    <use name="csdate"/>
</datePatterns>
</datetime>

Regex match exactly year, mont, day, hour and minute.
In props.conf I added

DATETIME_CONFIG = /etc/system/local/datetime.xml

SHOULD_LINEMERGE = FALSE

TIME_FORMAT = %Y%m%d%H%M

Any ideas why data are not indexed with resulting timestamp?
I tried to split in 2 regex, for timePatterns and datePatterns match, but the result is still the same.

Or do you suggest a different way to achieve timestamp override at index time?

Regards

1 Solution

abonuccelli_spl
Splunk Employee
Splunk Employee

Problem here I believe was due to actual timestamps in raw event past the default (MAX_TIMESTAMP_LOOKAHEAD) 150 chars.

View solution in original post

abonuccelli_spl
Splunk Employee
Splunk Employee

Problem here I believe was due to actual timestamps in raw event past the default (MAX_TIMESTAMP_LOOKAHEAD) 150 chars.

bizza
Path Finder

Yes, problem was MAX_TIMESTAMP_LOOKAHEAD.
Thanks for your help guys

ciao

0 Karma

abonuccelli_spl
Splunk Employee
Splunk Employee

MMM, well it works for me...
Bizza should be able to confirm
Antonio

0 Karma

marcoscala
Builder

Antonio,
I'm afraid but that's not the case either. In his/her case, date and time are splitted in the event data, so usual timeformat is more complex to manage. Unfortunately we can't use REGEX for TIME_FORMAT, otherwise that was the solution.

Marco

0 Karma

lguinn2
Legend

You don't need a custom datetime.xml - I wouldn't do it that way. It is complicated and unnecessary.

In props.conf all you should need is

SHOULD_LINEMERGE = FALSE
TIME_FORMAT = %Y%m%d%H%M

Assuming that your timestamp looks like

201405021209

If not, please comment with an example or two of the timestamp.

0 Karma

bizza
Path Finder

Yes, I added the --Y --m ecc only to show where timestamp fields are.
Ignore it and you'll have the original log line.

0 Karma

nitesh218ss
Communicator

Hi in my log event and filename date is not present i want give a fix date to log so what is do ?

0 Karma

lmyrefelt
Builder

Did you add the --Y and --m into the event example as an clarification ?
otherwise you could try;
TIME_PREFIX = \d{numberOfDigits}\s++
MAX_TIMESTAMP_LOOKAHEAD = 20
TIME_FORMAT = %Y%m%d
TIME_FORMAT = --Y%Y--m%m--d%d

0 Karma

bizza
Path Finder

The problem is that timestamp is splitted on every lines.
For example:

204023600511105443000 20140422--Y2014--m04--d180000000005.0600000000000041096125031ABDCE 81234567 ABDCE F & C 10024 ABDCE F & C 45399700123456789000000000.104023600582105443000 386511186636492--H15--M36PSBP

every line has 300 characters(digits), fields are position-sensitive.
I added --Y, --m, --d, --H and --M just before timestamp fields.

I believe that a custom datetime.xml is my only option.

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...