Getting Data In

Multiple timestap formats and subsecond with two digits only

Paolo_Prigione
Builder

Hi everybody, is it possible to teach a custom datetime.xml that my subsecond field is only two digit long?

I have a log file with two different timestamp formats, one of which is not automatically recognized. Here's an example:

00000000000000000001289525909952;2010-11-12 02:38:32:47;P2LIV ....

00000000000000000001289525916501;2010111202383591;YT00B ....

so I had to configure a custom datetime.xml which is working fine, except I am not able to let it extract the subseconds: Splunk always interprets them as zero.
Here's my custom datetime.xml:

<datetime>
<define name="_combdatetime_consMSG1" extract="year, month, day, hour, minute, second, subsecond">
    <!-- ...  2010-11-12 02:38:36:90 ...' -->
    <text><![CDATA[(\d+)-(\d+)-(\d+)\s(\d+):(\d+):(\d+):(\d+)]]></text>
</define>
<define name="_combdatetime_consMSG2" extract="year, month, day, hour, minute, second, subsecond">
    <!-- ...  2010111202383691 ...' -->
    <text><![CDATA[(\d{4})(\d{2})(\d{2})(\d{2})(\d{2})(\d{2})(\d{2})]]></text>
</define>
<timePatterns>
      <use name="_combdatetime_consMSG1"/>
      <use name="_combdatetime_consMSG2"/>
</timePatterns>
<datePatterns>
      <use name="_combdatetime_consMSG1"/>
      <use name="_combdatetime_consMSG2"/>
</datePatterns>
</datetime>

And this is how Splunk web displays those two timestamps

11/12/10 2:38:32.000 AM

11/12/10 2:38:35.000 AM

I've tried to manually modify the source file to put a trailing 0 after the timestamp (to bring it to a millis format) and changed the regexes accordingly, but had no luck (after a full restart, too) Anybody has an idea?

Thanks, Paolo

0 Karma

dwaddle
SplunkTrust
SplunkTrust

Hi Paolo,

You may be better off using TIME_FORMAT in props.conf rather than changing datetime.xml. Its handling of subseconds and subsecond precision is covered on another answers post:

http://answers.splunk.com/questions/1946/time-format-and-subseconds

dwaddle
SplunkTrust
SplunkTrust

D'oh! I completely missed that aspect of what you were explaining.

0 Karma

Paolo_Prigione
Builder

Thanks dwaddle, I've tried the %2N to specify only two digits (or 9, on other CICS logs), but then I would have to stick with one format only and "lose" the second.

0 Karma

Paolo_Prigione
Builder

As a sidenote: MAX_TIMESTAMP_LOOKAHEAD is set high enough for Splunk to read the whole timestamp.

0 Karma
Get Updates on the Splunk Community!

Detecting Remote Code Executions With the Splunk Threat Research Team

WATCH NOWRemote code execution (RCE) vulnerabilities pose a significant risk to organizations. If exploited, ...

Enter the Splunk Community Dashboard Challenge for Your Chance to Win!

The Splunk Community Dashboard Challenge is underway! This is your chance to showcase your skills in creating ...

.conf24 | Session Scheduler is Live!!

.conf24 is happening June 11 - 14 in Las Vegas, and we are thrilled to announce that the conference catalog ...