edit datetime.xml for my custom date and time in source field

[ips_mandar]

Hi everyone,
Can someone tell me what I'm suppose to edit in my datetime.xml file for my custom date and time to be recognized in Splunk?
I want to extract date and time from my source field which is like this-20190128T06:14:25.json IT is in format %Y%m%dT%H:%M:%S
I tried below but it won't help-

<define name="_masheddate4" extract="year, month, day">
    <text><![CDATA[(?:^|source::)^(\d{4})(\d{2})(\d{2})]]></text>
</define>
<define name="_mashedtime4" extract="hour, minute, second">
    <text><![CDATA[(?:^|source::)^.{9}(\d{2})]]></text>
        <use name="_hour"/>
    <text><![CDATA[(?:^|source::)^.{12}(\d{2})]]></text>
        <use name="_minute"/>
    <text><![CDATA[(?:^|source::)^.{15}(\d{2})]]></text>
        <use name="_second"/> 
</define>

in props.conf

[mysourcetype]
DATETIME_CONFIG = F:\Splunk\etc\apps\search\default\datetime.xml

My Splunk version is 7.1.2 and windows OS.

[woodcock]

If your distributing to indexers via a Cluster Master, use this:

\etc\slave-apps\search\default\datetime.xml

See here:
https://answers.splunk.com/answers/526680/splunk-ise-ta-fails-when-distributed-via-cluster-m.html

[ips_mandar]

Thanks for comment @woodcock
But I am not using cluster environment and using distributed environment and set props.conf in Heavy forwarder

[woodcock]

Check the error logs for datetime.xml for a hint.

[woodcock]

Try this:

<datetime>

<define name="_timeAndDateFromFilename_date" extract="year, month, day">
        <text><![CDATA[source::.*?(\d{4})-(\d{2})-(\d{2}T)]]></text>
</define>
<define name="_timeAndDateFromFilename_time" extract="hour, minute, second">
        <text><![CDATA[source::.*?T(\d{2}):(\d{2}):(\d{2})]]></text>
</define>

<timePatterns>
        <use name="_timeAndDateFromFilename_time"/>
</timePatterns>
<datePatterns>
        <use name="_timeAndDateFromFilename_date"/>
</datePatterns>

</datetime>

[ips_mandar]

Thanks @woodcock.. But My source file is for ex. 20190128T06:14:25.json .It is in format %Y%m%dT%H:%M:%S so I removed - as below but still it won't work.Not sure whats the issue.

<datetime>

 <define name="_timeAndDateFromFilename_date" extract="year, month, day">
         <text><![CDATA[source::.*?(\d{4})(\d{2})(\d{2}T)]]></text>
 </define>
 <define name="_timeAndDateFromFilename_time" extract="hour, minute, second">
         <text><![CDATA[source::.*?T(\d{2}):(\d{2}):(\d{2})]]></text>
 </define>

 <timePatterns>
         <use name="_timeAndDateFromFilename_time"/>
 </timePatterns>
 <datePatterns>
         <use name="_timeAndDateFromFilename_date"/>
 </datePatterns>

 </datetime>

[woodcock]

You also have this wrong:

DATETIME_CONFIG = F:\Splunk\etc\apps\search\default\datetime.xml

It should be this:

DATETIME_CONFIG = \etc\apps\search\default\datetime.xml

[ips_mandar]

thanks for comment @woodcock
Yes I have already changed DateTime_config to this -

DATETIME_CONFIG = /etc/system/local/datetime.xml

and placed datetime.xml file in system/local path.
but still unable to parse timestamp from source file.

[woodcock]

That is the wrong place for it and has changed the way that Splunk handles unknown events. It should NOT go there (at least not with that name). Did you try putting it where I told you to put it and using th exact props setting that I showed you?

[sowings]

1) Your regexes are wrong; the source:: is literally part of the field value, so another caret (^) is not going to have any meaning; there's no beginning of the line there.
2) The path to datetime.xml for the props.conf settings is always relative to $SPLUNK_HOME (or %SPLUNK_HOME%). The default value reads as /etc/datetime.xml for my install in /opt/splunk/; the full path to datetime.xml is /opt/splunk/etc/datetime.xml. You've provided a fully qualified path from the drive letter, you'll want to edit it down (looks like it should be /etc/apps/search/default/datetime.xml given the current setting you quoted).

[ips_mandar]

@sowings Thanks and I agree with you ..
i have already fixed props.conf with /etc/system/local/datetime.xml
and regarding regex I tried various combination but unable to succeed with any regex could you please help me to write regex to extract from source.
recently I tried below regex but unable to get success

<text><![CDATA[(?:source::).*?(?<!\d|\d\.|-)(?:20)?([901]\d)(0\d|1[012])([012]\d|3[01])(?!\d|-| {2,})[.T]?([01]\d|2[0123])[:]([0-6]\d)[:]([0-6]\d)(?:\.?(\d+))?]]></text>

[ips_mandar]

It seems to be bug to parse %Y%m%dT%H:%M:%S this format timestamp from source value

[sowings]

My mistake. I misread your initial problem statement. You're right that Splunk wants to assign a time to each and every event that it reads. Further, it processes the date and the time of events in separate passes. The way I've dealt with this in the past is to use DATETIME_CONFIG = CURRENT, which will take the "wall clock" time when the file is ingested as the time of the events. This should be within a second or two of when it gets generated, typically.

It seems that upgrading is out of the question for you?

[lakshman239]

Did you try the INGEST_EVAL mentioned in the bottom of the answers at https://answers.splunk.com/answers/320978/how-to-extract-the-timestamp-from-a-filename-at-in.html ?

You may also want to check https://www.function1.com/2013/01/oh-no-splunking-log-files-with-multiple-formats-no-problem

[ips_mandar]

Hi @lakshman239,
INGEST_EVAL is for recent version but my version is 7.1.2 where it is not applicable.
Also I tried below-

<datetime>

<!-- [2012/06/01 8:54:21.599] 20190128T06:14:25.json-->
<define name="_datetimeformat2" extract="year, month, day, hour, minute, second">
<text><![CDATA[(?:^|source::)^(\d{4})(\d{2})(\d{2})T(\d{2}):(\d{2}):(\d{2})]]</text>
</define>

<timePatterns>

<use name="_datetimeformat2"/>

</timePatterns>
<datePatterns>

<use name="_datetimeformat2"/>

</datePatterns>
</datetime>

but still I am getting error as-
Uncaught exception in Aggregator, skipping an event: Error parsing regex XML file: F:\Splunk\etc\apps\search\default\datetime.xml - Couldn't find 'timePatterns' in config data for AggregatorProcessor. - data_source="20190128T16:15:23.json

[petom]

Try to use the following regex:

<text><![CDATA[^(?:|source::)(\d{4})(\d{2})(\d{2})T(\d{2}):(\d{2}):(\d{2})]]</text>

That should fix the error, but it will still not populate timestamp (_time field) from the source filename.

[ips_mandar]

Thanks for comment @petom . But it won't help to populate Timestamp from source..I also tried below but it also won't help-

<text><![CDATA[(?:source::).*?(?<!\d|\d\.|-)(?:20)?([901]\d)(0\d|1[012])([012]\d|3[01])(?!\d|-| {2,})[.T]?([01]\d|2[0123])[:]([0-6]\d)[:]([0-6]\d)(?:\.?(\d+))?]]></text>

I tried various combination but none of them help to populate timestamp from source .

[petom]

No, it won't help. I said that in my comment. You need to upgrade to version 7.2 and that opens door to get the timestamp from source.