Splunk Search

edit datetime.xml for my custom date and time in source field

Contributor

Hi everyone,
Can someone tell me what I'm suppose to edit in my datetime.xml file for my custom date and time to be recognized in Splunk?
I want to extract date and time from my source field which is like this-20190128T06:14:25.json IT is in format %Y%m%dT%H:%M:%S
I tried below but it won't help-

<define name="_masheddate4" extract="year, month, day">
    <text><![CDATA[(?:^|source::)^(\d{4})(\d{2})(\d{2})]]></text>
</define>
<define name="_mashedtime4" extract="hour, minute, second">
    <text><![CDATA[(?:^|source::)^.{9}(\d{2})]]></text>
        <use name="_hour"/>
    <text><![CDATA[(?:^|source::)^.{12}(\d{2})]]></text>
        <use name="_minute"/>
    <text><![CDATA[(?:^|source::)^.{15}(\d{2})]]></text>
        <use name="_second"/> 
</define>

in props.conf

[mysourcetype]
DATETIME_CONFIG = F:\Splunk\etc\apps\search\default\datetime.xml

My Splunk version is 7.1.2 and windows OS.

0 Karma

Esteemed Legend

If your distributing to indexers via a Cluster Master, use this:

\etc\slave-apps\search\default\datetime.xml

See here:
https://answers.splunk.com/answers/526680/splunk-ise-ta-fails-when-distributed-via-cluster-m.html

0 Karma

Contributor

Thanks for comment @woodcock
But I am not using cluster environment and using distributed environment and set props.conf in Heavy forwarder

0 Karma

Esteemed Legend

Check the error logs for datetime.xml for a hint.

0 Karma

Esteemed Legend

Try this:

<datetime>

<define name="_timeAndDateFromFilename_date" extract="year, month, day">
        <text><![CDATA[source::.*?(\d{4})-(\d{2})-(\d{2}T)]]></text>
</define>
<define name="_timeAndDateFromFilename_time" extract="hour, minute, second">
        <text><![CDATA[source::.*?T(\d{2}):(\d{2}):(\d{2})]]></text>
</define>

<timePatterns>
        <use name="_timeAndDateFromFilename_time"/>
</timePatterns>
<datePatterns>
        <use name="_timeAndDateFromFilename_date"/>
</datePatterns>

</datetime>
0 Karma

Contributor

Thanks @woodcock.. But My source file is for ex. 20190128T06:14:25.json .It is in format %Y%m%dT%H:%M:%S so I removed - as below but still it won't work.Not sure whats the issue.

<datetime>

 <define name="_timeAndDateFromFilename_date" extract="year, month, day">
         <text><![CDATA[source::.*?(\d{4})(\d{2})(\d{2}T)]]></text>
 </define>
 <define name="_timeAndDateFromFilename_time" extract="hour, minute, second">
         <text><![CDATA[source::.*?T(\d{2}):(\d{2}):(\d{2})]]></text>
 </define>

 <timePatterns>
         <use name="_timeAndDateFromFilename_time"/>
 </timePatterns>
 <datePatterns>
         <use name="_timeAndDateFromFilename_date"/>
 </datePatterns>

 </datetime>
0 Karma

Esteemed Legend

You also have this wrong:

DATETIME_CONFIG = F:\Splunk\etc\apps\search\default\datetime.xml

It should be this:

DATETIME_CONFIG = \etc\apps\search\default\datetime.xml
0 Karma

Contributor

thanks for comment @woodcock
Yes I have already changed DateTime_config to this -

DATETIME_CONFIG = /etc/system/local/datetime.xml

and placed datetime.xml file in system/local path.
but still unable to parse timestamp from source file.

0 Karma

Esteemed Legend

That is the wrong place for it and has changed the way that Splunk handles unknown events. It should NOT go there (at least not with that name). Did you try putting it where I told you to put it and using th exact props setting that I showed you?

0 Karma

Splunk Employee
Splunk Employee

1) Your regexes are wrong; the source:: is literally part of the field value, so another caret (^) is not going to have any meaning; there's no beginning of the line there.
2) The path to datetime.xml for the props.conf settings is always relative to $SPLUNKHOME (or %SPLUNKHOME%). The default value reads as /etc/datetime.xml for my install in /opt/splunk/; the full path to datetime.xml is /opt/splunk/etc/datetime.xml. You've provided a fully qualified path from the drive letter, you'll want to edit it down (looks like it should be /etc/apps/search/default/datetime.xml given the current setting you quoted).

0 Karma

Contributor

@sowings Thanks and I agree with you ..
i have already fixed props.conf with /etc/system/local/datetime.xml
and regarding regex I tried various combination but unable to succeed with any regex could you please help me to write regex to extract from source.
recently I tried below regex but unable to get success

<text><![CDATA[(?:source::).*?(?<!\d|\d\.|-)(?:20)?([901]\d)(0\d|1[012])([012]\d|3[01])(?!\d|-| {2,})[.T]?([01]\d|2[0123])[:]([0-6]\d)[:]([0-6]\d)(?:\.?(\d+))?]]></text>
0 Karma

Contributor

It seems to be bug to parse %Y%m%dT%H:%M:%S this format timestamp from source value

0 Karma

Splunk Employee
Splunk Employee

My mistake. I misread your initial problem statement. You're right that Splunk wants to assign a time to each and every event that it reads. Further, it processes the date and the time of events in separate passes. The way I've dealt with this in the past is to use DATETIME_CONFIG = CURRENT, which will take the "wall clock" time when the file is ingested as the time of the events. This should be within a second or two of when it gets generated, typically.

It seems that upgrading is out of the question for you?

0 Karma

SplunkTrust
SplunkTrust
0 Karma

Contributor

Hi @lakshman239,
INGEST_EVAL is for recent version but my version is 7.1.2 where it is not applicable.
Also I tried below-

<datetime>

<!-- [2012/06/01 8:54:21.599] 20190128T06:14:25.json-->
<define name="_datetimeformat2" extract="year, month, day, hour, minute, second">
<text><![CDATA[(?:^|source::)^(\d{4})(\d{2})(\d{2})T(\d{2}):(\d{2}):(\d{2})]]</text>
</define>

<timePatterns>

<use name="_datetimeformat2"/>

</timePatterns>
<datePatterns>

<use name="_datetimeformat2"/>

</datePatterns>
</datetime>

but still I am getting error as-
Uncaught exception in Aggregator, skipping an event: Error parsing regex XML file: F:\Splunk\etc\apps\search\default\datetime.xml - Couldn't find 'timePatterns' in config data for AggregatorProcessor. - data_source="20190128T16:15:23.json

0 Karma

Path Finder

Try to use the following regex:

<text><![CDATA[^(?:|source::)(\d{4})(\d{2})(\d{2})T(\d{2}):(\d{2}):(\d{2})]]</text>

That should fix the error, but it will still not populate timestamp (_time field) from the source filename.

0 Karma

Contributor

Thanks for comment @petom . But it won't help to populate Timestamp from source..I also tried below but it also won't help-

<text><![CDATA[(?:source::).*?(?<!\d|\d\.|-)(?:20)?([901]\d)(0\d|1[012])([012]\d|3[01])(?!\d|-| {2,})[.T]?([01]\d|2[0123])[:]([0-6]\d)[:]([0-6]\d)(?:\.?(\d+))?]]></text>

I tried various combination but none of them help to populate timestamp from source .

0 Karma

Path Finder

No, it won't help. I said that in my comment. You need to upgrade to version 7.2 and that opens door to get the timestamp from source.

0 Karma