Getting Data In

custom datetime.xml is not working for me

asimagu
Builder

hi guys

I have 2 different kind of events inside the same file. I am aware that I need to use a custom datetime.xml in order to recognize the timestamp of each sort of event. However it is not working for me.

These are the two kinds of events and I highlight in bold what should be recognized as the timestamp:

1081|2|20150512|436959|1660|0.00|1.00|0.00|4.5000|0|20

1081|2|436968|20150512|20150512|1336|1|1|000|0.00|0.00|3.20|5959|0034|G|0|20

This is what I coded in my new datetime.xml

<datetime>

<define name="_mc1" extract="year, month, day">
    &lt;![CDATA[^[^\|]*(\|[^\|]*)\|(\d\d\d\d)(\d\d)(\d\d)(\|[^\|]*){8}$]]&gt;
</define>
<define name="_mc2" extract="year, month, day, hour, minute">
    &lt;![CDATA[^[^\|]*(\|[^\|]*){3}\|(\d\d\d\d)(\d\d)(\d\d)\|(\d\d)(\d\d)(\|[^\|]*){11}$]]&gt;
</define>
<timePatterns>
     <use name="_mc2"/>
</timePatterns>
<datePatterns>
     <use name="_mc1"/>
     <use name="_mc2"/>
</datePatterns>
</datetime>

and in my props.conf this is what I have:

[test]
SHOULD_LINEMERGE = false
MAX_EVENTS = 1
DATETIME_CONFIG = /etc/apps/testing/local/datetime.xml
TZ = Europe/Madrid
MAX_TIMESTAMP_LOOKAHEAD=100

any ideas why my code is not working??

Tags (2)
1 Solution

asimagu
Builder

So, I happened to find the solution and the source of the problem:

When doing the regex, every parenthesis is a capture group that will go into each of the timestamp variables.

I tried using ?: at the groups that I did not need to remember/capture and voila!! it works!!

This is it:

<datetime>

<define name="_mc1" extract="year, month, day">
    <text><![CDATA[^[^\|]*(?:\|[^\|]*)\|(\d\d\d\d)(\d\d)(\d\d)(?:\|[^\|]*){8}$]]></text>
</define>
<define name="_mc2" extract="year, month, day, hour, minute">
    <text><![CDATA[^[^\|]*(?:\|[^\|]*){3}\|(\d\d\d\d)(\d\d)(\d\d)\|(\d\d)(\d\d)(?:\|[^\|]*){11}$]]></text>
</define>
<timePatterns>
     <use name="_mc2"/>
</timePatterns>
<datePatterns>
     <use name="_mc1"/>
     <use name="_mc2"/>
</datePatterns>
</datetime>

View solution in original post

asimagu
Builder

So, I happened to find the solution and the source of the problem:

When doing the regex, every parenthesis is a capture group that will go into each of the timestamp variables.

I tried using ?: at the groups that I did not need to remember/capture and voila!! it works!!

This is it:

<datetime>

<define name="_mc1" extract="year, month, day">
    <text><![CDATA[^[^\|]*(?:\|[^\|]*)\|(\d\d\d\d)(\d\d)(\d\d)(?:\|[^\|]*){8}$]]></text>
</define>
<define name="_mc2" extract="year, month, day, hour, minute">
    <text><![CDATA[^[^\|]*(?:\|[^\|]*){3}\|(\d\d\d\d)(\d\d)(\d\d)\|(\d\d)(\d\d)(?:\|[^\|]*){11}$]]></text>
</define>
<timePatterns>
     <use name="_mc2"/>
</timePatterns>
<datePatterns>
     <use name="_mc1"/>
     <use name="_mc2"/>
</datePatterns>
</datetime>

laserval
Communicator

The path /etc/apps/testing/local/datetime.xml looks like an absolute path from the filesystem root. Does changing it to etc/apps/testing/local/datetime.xml help?

0 Karma

asimagu
Builder

no, same thing. The location is not the problem

0 Karma
Get Updates on the Splunk Community!

AppDynamics Summer Webinars

This summer, our mighty AppDynamics team is cooking up some delicious content on YouTube Live to satiate your ...

SOCin’ it to you at Splunk University

Splunk University is expanding its instructor-led learning portfolio with dedicated Security tracks at .conf25 ...

Credit Card Data Protection & PCI Compliance with Splunk Edge Processor

Organizations handling credit card transactions know that PCI DSS compliance is both critical and complex. The ...