Getting Data In
Highlighted

How can we set TIME_FORMAT in props.conf where the milliseconds vary in length?

Ultra Champion

We are trying to create a TIME_FORMAT where the milliseconds vary in length. Sometimes it is two digits and sometime it is three digits. For example:

 2018-09-11 04:28:05.14
 2018-09-11 04:20:55.336

What can we do?

We have something like %Y-%m-%d %H:%M:%S.%2N but that won't work if the milliseconds are three in length.

0 Karma
Highlighted

Re: How can we set TIME_FORMAT in props.conf where the milliseconds vary in length?

Ultra Champion

Our Sales Engineer said -

-- I mean, the correct way is to fix the logs. Drifting time formats is pretty awful, and would usually indicate there should either be 2 log files or a problem in the code.

Otherwise, just set the TIME_PREFIX and let Splunk do the normal timestamp magic. Both should be automatically parsed, but test that first.

0 Karma
Highlighted

Re: How can we set TIME_FORMAT in props.conf where the milliseconds vary in length?

SplunkTrust
SplunkTrust

Hi @ddrillic,

You can use custom datetime.xml for this type of different timestamp, ref. documentation http://docs.splunk.com/Documentation/SplunkCloud/latest/Data/Configuredatetimexml

If your source type contains only those 2 types of date/time format, then you can use the below configuration.

For testing purpose I have created datetime.xml in $SPLUNK_HOME/etc/apps/search/local/ with below configurations

<!--   Version 4.0 -->

<!-- datetime.xml -->
<!-- This file contains the general formulas for parsing date/time formats. -->

<datetime>

<define name="test_1_date" extract="year, month, day">
        <text><![CDATA[(\d+)-(\d+)-(\d+)]]></text>
</define>

<define name="test_1_time" extract="hour, minute, second, subsecond">
        <text><![CDATA[-\d+-\d+\s(\d+):(\d+):(\d+)\.(\d+)]]></text>
</define>

<timePatterns>
      <use name="test_1_time"/>
</timePatterns>
<datePatterns>
      <use name="test_1_date"/>
</datePatterns>

</datetime>

props.conf in $SPLUNK_HOME/etc/apps/search/local/ with below configurations

[test_sourcetype]
DATETIME_CONFIG = /etc/apps/search/local/datetime.xml
NO_BINARY_CHECK = true
disabled = false
pulldown_type = true
MAX_TIMESTAMP_LOOKAHEAD = 23

Restarted splunk and ingested sample data and it is extracting timestamp as 2018-09-11 04:28:05.140 (Here splunk is assuming subsecond as 140 not 014) and 2018-09-11 04:20:55.336, you can give it try in your test environment.

EDIT: Updated props.conf config (Added MAX_TIMESTAMP_LOOKAHEAD)

View solution in original post

0 Karma
Highlighted

Re: How can we set TIME_FORMAT in props.conf where the milliseconds vary in length?

Ultra Champion

Gorgeous answer @harsmarvania57 - much appreciated.

0 Karma
Speak Up for Splunk Careers!

We want to better understand the impact Splunk experience and expertise has has on individuals' careers, and help highlight the growing demand for Splunk skills.