topic Re: Hex Time Stamp Extraction issues with DATETIME_CONFIG in Dashboards & Visualizations

Hex Time Stamp Extraction issues with DATETIME_CONFIG

dmaislin_splunk — Mon, 19 Sep 2011 20:04:06 GMT

I have a log file from EMC where the timestamp is in hex:  timeStamp="0x4E665CB6000269D7".  I have provided a sample log snippet and all the .conf files below.  I can't get it to recognize the hex time stamp.  What am I doing wrong?

log file:

CheckEvent: Request  : <CheckEventRequest>
  <EventList count="1">
    <Event event="0x100000" path="\\omega22.w2k8r2.vee.com\CHECK$\server2fs1\test" flag="0x2" protocol="0" server="OMEGA22" share="server2fs1" clientIP="10.245.72.115" serverIP="10.245.72.209" timeStamp="0x4E665CB6000269D7" userSid="S-1-5-21-3594340890-534397530-1661201549-500" ownerSid="S-1-5-21-3594340890-534397530-1661201549-500" fileSize="0x0" desiredAccess="0x100081" createDispo="0x1" ntStatus="0x0" relativePath="\\OMEGA22\server2fs1\test"/>
  </EventList>
</CheckEventRequest>

inputs.conf

[monitor:///Users/dmaislin/Desktop/EMC/fsc.txt]
disabled = false
followTail = 0
sourcetype = emc

props.conf

[source::/Users/dmaislin/Desktop/EMC/fsc.txt]
sourcetype=emc

[emc]
TIME_PREFIX = timeStamp
MAX_TIMESTAMP_LOOKAHEAD = 9999
BREAK_ONLY_BEFORE = CheckEvent:
MUST_NOT_BREAK_BEFORE = </CheckEventRequest>
SHOULD_LINEMERGE = TRUE
DATETIME_CONFIG = /etc/system/local/emc-epoch.xml
REPORT-xmlkv = xmlkv-alternative

transforms.conf

[xmlkv-alternative]
REGEX = <([^\s\>]*)[^\>]*\>([^<]*)\<\/\1\>
FORMAT = $1::$2
MV_ADD = True

emc-epoch.xml

<datetime>
         <define name="_hexepoch" extract="hexepoch">
           <text><![CDATA[timeStamp="0x([A-Fa-f0-9]{8})]]></text>
         </define>
         <timePatterns>
           <use name="_hexepoch"/>
         </timePatterns>
         <datePatterns>
         </datePatterns>
</datetime>

Re: Hex Time Stamp Extraction issues with DATETIME_CONFIG

hexx — Mon, 28 Sep 2020 09:53:25 GMT

The 1st problem I can see here is that the source-based stanza [source:///Users/dmaislin/Desktop/EMC/fsc.txt] will take precedence over the sourcetype-based [emc] stanza.

As per props.conf.spec :

**[<spec>] stanza precedence:**

For settings that are specified in multiple categories of matching [<spec>] stanzas,
[host::<host>] settings override [<sourcetype>] settings. Additionally,
[source::<source>] settings override both [host::<host>] and
[<sourcetype>] settings.

You should not have two stanzas here.

Also :

Your TIME_PREFIX should be :

TIME_PREFIX = timeStamp="

...and to reflect this, I would suggest that you change your time-stamp extraction regex in emc-epoch.xml with :

<![CDATA[0x([A-Fa-f0-9]{8})]]>

Your MAX_TIMESTAMP_LOOKAHEAD should be anchored to the end of TIME_PREFIX, not to the beginning of the event :

MAX_TIMESTAMP_LOOKAHEAD = 16

Re: Hex Time Stamp Extraction issues with DATETIME_CONFIG

dmaislin_splunk — Mon, 19 Sep 2011 23:55:37 GMT

SOLVED. Thanks! All configs are updated above.

Re: Hex Time Stamp Extraction issues with DATETIME_CONFIG

hexx — Tue, 20 Sep 2011 00:12:48 GMT

Aw, now we can't see what the initial problem was! Don't be mad but I'll revert your question to the version that shows them.

Re: Hex Time Stamp Extraction issues with DATETIME_CONFIG

dmaislin_splunk — Tue, 20 Sep 2011 12:58:24 GMT

Here are the fixed files...

inputs.conf

[monitor:///Users/dmaislin/Desktop/EMC/fsc.txt]
disabled = false
followTail = 0
sourcetype = emc

props.conf

[source::/Users/dmaislin/Desktop/EMC/fsc.txt]
TIME_PREFIX = timeStamp="
MAX_TIMESTAMP_LOOKAHEAD = 16
BREAK_ONLY_BEFORE = CheckEvent:
MUST_NOT_BREAK_BEFORE = </CheckEventRequest>
SHOULD_LINEMERGE = TRUE
DATETIME_CONFIG = /etc/system/local/emc-epoch.xml
REPORT-xmlkv = xmlkv-alternative
sourcetype = emc

transforms.conf

[xmlkv-alternative]
REGEX = <([^\s\>]*)[^\>]*\>([^<]*)\<\/\1\>
FORMAT = $1::$2
MV_ADD = True

emc-epoch.xml

<datetime>
         <define name="_hexepoch" extract="hexepoch">
           <text><![CDATA[0x([A-Fa-f0-9]{8})]]></text>
         </define>
         <timePatterns>
           <use name="_hexepoch"/>
         </timePatterns>
         <datePatterns>
         </datePatterns>
</datetime>