Getting Data In

TIME_FORMAT after overrided source type on a per-event basis

bever
Explorer

Hello,

I have a file exampleFile that has two different timestamp/event formats:

~02 07 10:19:24 OIT-FO-OFR2 NSSTRAP 

and

Feb 05 18:58:43 ANSU-OPS-2 checkpn: OK:ABM3CANDAF34018

As both timestamps do not contain the year, splunk does not manage to correctly index the events.

I therefore override both sourcetypes on a per-event basis.

In props.conf:

[source::.../exampleFile]
TRANSFORMS-event_1 = event_1
TRANSFORMS-event_2 = event_2

[FORMAT_1]
NO_BINARY_CHECK = 1
TIME_FORMAT =%b %d %H:%M:%S

[FORMAT_2]
NO_BINARY_CHECK = 1
TIME_PREFIX =^\~
TIME_FORMAT =%m %d %H:%M:%S

In transforms.conf:

[event_1]
REGEX = \w{3}\s\d{2}\s\d{2}\:\d{2}\:\d{2}\s.+
FORMAT = sourcetype::FORMAT_1
DEST_KEY = MetaData:Sourcetype

[event_2]
REGEX = \~\d{2}\s\d{2}\s\d{2}\:\d{2}\:\d{2}\s.+
FORMAT = sourcetype::FORMAT_2
DEST_KEY = MetaData:Sourcetype

This works, the sourcetype is correctly assigned to each type, but the indexed timestamps stay wrong.

Any ideas on how I can correctly assign the TIME_FORMAT to the per-event overrided sourcetype?

PS: When I upload a file only containing one event format, and when I assign this file directly to a sourcetype FORMAT_1 or FORMAT_2, the TIME_FORMAT definition works correctly

0 Karma
1 Solution

martin_mueller
SplunkTrust
SplunkTrust

To avoid that order issue altogether you can do this in props.conf:

[both_formats]
DATETIME_CONFIG=/etc/system/local/mydatetime.xml
NO_BINARY_CHECK=1
SHOULD_LINEMERGE=false

Apply that sourcetype to the entire file, no transforms.conf shenanigans. The content of mydatetime.xml is as follows:

<datetime>
  <define name="_year" extract="year">
    <text><![CDATA[(20\d\d|19\d\d|[901]\d(?!\d))]]></text>
  </define>
  <define name="_month" extract="month">
    <text><![CDATA[(0?[1-9]|1[012])(?!:)]]></text>
  </define>
  <define name="_litmonth"  extract="litmonth">
    <text><![CDATA[(?<![\d\w])(jan|\x{3127}\x{6708}|feb|\x{4E8C}\x{6708}|mar|\x{4E09}\x{6708}|apr|\x{56DB}\x{6708}|may|\x{4E94}\x{6708}|jun|\x{516D}\x{6708}|jul|\x{4E03}\x{6708}|aug|\x{516B}\x{6708}|sep|\x{4E5D}\x{6708}|oct|\x{5341}\x{6708}|nov|\x{5341}\x{3127}\x{6708}|dec|\x{5341}\x{4E8C}\x{6708})[a-z,\.;]*]]></text>
  </define>
  <define name="_day"  extract="day">
    <text><![CDATA[(0?[1-9]|[12]\d|3[01])]]></text> 
  </define>
  <define name="_hour" extract="hour">
    <text><![CDATA[([01]?[1-9]|[012][0-3])(?!\d)]]></text>
  </define>
  <define name="_minute" extract="minute">
    <text><![CDATA[([0-6]\d)(?!\d)]]></text>
  </define>
  <define name="_second" extract="second">
    <text><![CDATA[([0-6]\d)(?!\d)]]></text>
  </define>
  <define name="format_1" extract="litmonth, day, hour, minute, second">
    <text><![CDATA[(\w\w\w) (\d\d?) (\d\d):(\d\d):(\d\d)]]></text>
  </define>
  <define name="format_2" extract="month, day, hour, minute, second">
    <text><![CDATA[(\d\d?) (\d\d?) (\d\d):(\d\d):(\d\d)]]></text>
  </define>
  <timePatterns>
    <use name="format_1"/>
    <use name="format_2"/>
  </timePatterns>
  <datePatterns>
  </datePatterns>
</datetime>

Take a look at $SPLUNK_HOME/etc/datetime.xml for the default version.

View solution in original post

martin_mueller
SplunkTrust
SplunkTrust

To avoid that order issue altogether you can do this in props.conf:

[both_formats]
DATETIME_CONFIG=/etc/system/local/mydatetime.xml
NO_BINARY_CHECK=1
SHOULD_LINEMERGE=false

Apply that sourcetype to the entire file, no transforms.conf shenanigans. The content of mydatetime.xml is as follows:

<datetime>
  <define name="_year" extract="year">
    <text><![CDATA[(20\d\d|19\d\d|[901]\d(?!\d))]]></text>
  </define>
  <define name="_month" extract="month">
    <text><![CDATA[(0?[1-9]|1[012])(?!:)]]></text>
  </define>
  <define name="_litmonth"  extract="litmonth">
    <text><![CDATA[(?<![\d\w])(jan|\x{3127}\x{6708}|feb|\x{4E8C}\x{6708}|mar|\x{4E09}\x{6708}|apr|\x{56DB}\x{6708}|may|\x{4E94}\x{6708}|jun|\x{516D}\x{6708}|jul|\x{4E03}\x{6708}|aug|\x{516B}\x{6708}|sep|\x{4E5D}\x{6708}|oct|\x{5341}\x{6708}|nov|\x{5341}\x{3127}\x{6708}|dec|\x{5341}\x{4E8C}\x{6708})[a-z,\.;]*]]></text>
  </define>
  <define name="_day"  extract="day">
    <text><![CDATA[(0?[1-9]|[12]\d|3[01])]]></text> 
  </define>
  <define name="_hour" extract="hour">
    <text><![CDATA[([01]?[1-9]|[012][0-3])(?!\d)]]></text>
  </define>
  <define name="_minute" extract="minute">
    <text><![CDATA[([0-6]\d)(?!\d)]]></text>
  </define>
  <define name="_second" extract="second">
    <text><![CDATA[([0-6]\d)(?!\d)]]></text>
  </define>
  <define name="format_1" extract="litmonth, day, hour, minute, second">
    <text><![CDATA[(\w\w\w) (\d\d?) (\d\d):(\d\d):(\d\d)]]></text>
  </define>
  <define name="format_2" extract="month, day, hour, minute, second">
    <text><![CDATA[(\d\d?) (\d\d?) (\d\d):(\d\d):(\d\d)]]></text>
  </define>
  <timePatterns>
    <use name="format_1"/>
    <use name="format_2"/>
  </timePatterns>
  <datePatterns>
  </datePatterns>
</datetime>

Take a look at $SPLUNK_HOME/etc/datetime.xml for the default version.

martin_mueller
SplunkTrust
SplunkTrust

Timestamping happens before the transforms rules, so the sourcetype gets set too late.

Some quite informative diagrams are here: http://wiki.splunk.com/Community:HowIndexingWorks

Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

Splunk is officially part of Cisco

Revolutionizing how our customers build resilience across their entire digital footprint.   Splunk ...

Splunk APM & RUM | Planned Maintenance March 26 - March 28, 2024

There will be planned maintenance for Splunk APM and RUM between March 26, 2024 and March 28, 2024 as ...