topic Re: edit datetime.xml for my custom date and time in source field in Splunk Search

edit datetime.xml for my custom date and time in source field

ips_mandar — Mon, 28 Jan 2019 13:52:13 GMT

Hi everyone,
Can someone tell me what I'm suppose to edit in my datetime.xml file for my custom date and time to be recognized in Splunk?
I want to extract date and time from my source field which is like this-20190128T06:14:25.json IT is in format %Y%m%dT%H:%M:%S
I tried below but it won't help-

<define name="_masheddate4" extract="year, month, day">
    <text><![CDATA[(?:^|source::)^(\d{4})(\d{2})(\d{2})]]></text>
</define>
<define name="_mashedtime4" extract="hour, minute, second">
    <text><![CDATA[(?:^|source::)^.{9}(\d{2})]]></text>
        <use name="_hour"/>
    <text><![CDATA[(?:^|source::)^.{12}(\d{2})]]></text>
        <use name="_minute"/>
    <text><![CDATA[(?:^|source::)^.{15}(\d{2})]]></text>
        <use name="_second"/> 
</define>

in props.conf

[mysourcetype]
DATETIME_CONFIG = F:\Splunk\etc\apps\search\default\datetime.xml

My Splunk version is 7.1.2 and windows OS.

Re: edit datetime.xml for my custom date and time in source field

lakshman239 — Mon, 28 Jan 2019 14:23:15 GMT

Did you try the INGEST_EVAL mentioned in the bottom of the answers at https://answers.splunk.com/answers/320978/how-to-extract-the-timestamp-from-a-filename-at-in.html ?

You may also want to check https://www.function1.com/2013/01/oh-no-splunking-log-files-with-multiple-formats-no-problem

Re: edit datetime.xml for my custom date and time in source field

ips_mandar — Mon, 28 Jan 2019 16:21:49 GMT

Hi @lakshman239,
INGEST_EVAL is for recent version but my version is 7.1.2 where it is not applicable.
Also I tried below-

<datetime>

<!-- [2012/06/01 8:54:21.599] 20190128T06:14:25.json-->
<define name="_datetimeformat2" extract="year, month, day, hour, minute, second">
<text><![CDATA[(?:^|source::)^(\d{4})(\d{2})(\d{2})T(\d{2}):(\d{2}):(\d{2})]]</text>
</define>

<timePatterns>

<use name="_datetimeformat2"/>

</timePatterns>
<datePatterns>

<use name="_datetimeformat2"/>

</datePatterns>
</datetime>

but still I am getting error as-
Uncaught exception in Aggregator, skipping an event: Error parsing regex XML file: F:\Splunk\etc\apps\search\default\datetime.xml - Couldn't find 'timePatterns' in config data for AggregatorProcessor. - data_source="20190128T16:15:23.json

Re: edit datetime.xml for my custom date and time in source field

petom — Wed, 30 Jan 2019 18:44:57 GMT

Try to use the following regex:

<text><![CDATA[^(?:|source::)(\d{4})(\d{2})(\d{2})T(\d{2}):(\d{2}):(\d{2})]]</text>

That should fix the error, but it will still not populate timestamp (_time field) from the source filename.

Re: edit datetime.xml for my custom date and time in source field

ips_mandar — Thu, 31 Jan 2019 07:59:00 GMT

Thanks for comment @petom . But it won't help to populate Timestamp from source..I also tried below but it also won't help-

<text><![CDATA[(?:source::).*?(?<!\d|\d\.|-)(?:20)?([901]\d)(0\d|1[012])([012]\d|3[01])(?!\d|-| {2,})[.T]?([01]\d|2[0123])[:]([0-6]\d)[:]([0-6]\d)(?:\.?(\d+))?]]></text>

I tried various combination but none of them help to populate timestamp from source .

Re: edit datetime.xml for my custom date and time in source field

sowings — Tue, 29 Sep 2020 23:02:47 GMT

1) Your regexes are wrong; the source:: is literally part of the field value, so another caret (^) is not going to have any meaning; there's no beginning of the line there.
2) The path to datetime.xml for the props.conf settings is always relative to $SPLUNK_HOME (or %SPLUNK_HOME%). The default value reads as /etc/datetime.xml for my install in /opt/splunk/; the full path to datetime.xml is /opt/splunk/etc/datetime.xml. You've provided a fully qualified path from the drive letter, you'll want to edit it down (looks like it should be /etc/apps/search/default/datetime.xml given the current setting you quoted).

Re: edit datetime.xml for my custom date and time in source field

ips_mandar — Thu, 31 Jan 2019 15:16:52 GMT

@sowings Thanks and I agree with you ..
i have already fixed props.conf with /etc/system/local/datetime.xml
and regarding regex I tried various combination but unable to succeed with any regex could you please help me to write regex to extract from source.
recently I tried below regex but unable to get success

<text><![CDATA[(?:source::).*?(?<!\d|\d\.|-)(?:20)?([901]\d)(0\d|1[012])([012]\d|3[01])(?!\d|-| {2,})[.T]?([01]\d|2[0123])[:]([0-6]\d)[:]([0-6]\d)(?:\.?(\d+))?]]></text>

Re: edit datetime.xml for my custom date and time in source field

ips_mandar — Thu, 31 Jan 2019 15:18:15 GMT

It seems to be bug to parse %Y%m%dT%H:%M:%S this format timestamp from source value

Re: edit datetime.xml for my custom date and time in source field

sowings — Fri, 01 Feb 2019 18:56:27 GMT

My mistake. I misread your initial problem statement. You're right that Splunk wants to assign a time to each and every event that it reads. Further, it processes the date and the time of events in separate passes. The way I've dealt with this in the past is to use DATETIME_CONFIG = CURRENT, which will take the "wall clock" time when the file is ingested as the time of the events. This should be within a second or two of when it gets generated, typically.

It seems that upgrading is out of the question for you?

Re: edit datetime.xml for my custom date and time in source field

petom — Fri, 01 Feb 2019 20:35:16 GMT

No, it won't help. I said that in my comment. You need to upgrade to version 7.2 and that opens door to get the timestamp from source.

Re: edit datetime.xml for my custom date and time in source field

woodcock — Sat, 02 Feb 2019 04:07:56 GMT

Try this:

<datetime>

<define name="_timeAndDateFromFilename_date" extract="year, month, day">
        <text><![CDATA[source::.*?(\d{4})-(\d{2})-(\d{2}T)]]></text>
</define>
<define name="_timeAndDateFromFilename_time" extract="hour, minute, second">
        <text><![CDATA[source::.*?T(\d{2}):(\d{2}):(\d{2})]]></text>
</define>

<timePatterns>
        <use name="_timeAndDateFromFilename_time"/>
</timePatterns>
<datePatterns>
        <use name="_timeAndDateFromFilename_date"/>
</datePatterns>

</datetime>

Re: edit datetime.xml for my custom date and time in source field

ips_mandar — Mon, 04 Feb 2019 15:26:21 GMT

Thanks @woodcock.. But My source file is for ex. 20190128T06:14:25.json .It is in format %Y%m%dT%H:%M:%S so I removed - as below but still it won't work.Not sure whats the issue.

<datetime>

 <define name="_timeAndDateFromFilename_date" extract="year, month, day">
         <text><![CDATA[source::.*?(\d{4})(\d{2})(\d{2}T)]]></text>
 </define>
 <define name="_timeAndDateFromFilename_time" extract="hour, minute, second">
         <text><![CDATA[source::.*?T(\d{2}):(\d{2}):(\d{2})]]></text>
 </define>

 <timePatterns>
         <use name="_timeAndDateFromFilename_time"/>
 </timePatterns>
 <datePatterns>
         <use name="_timeAndDateFromFilename_date"/>
 </datePatterns>

 </datetime>

Re: edit datetime.xml for my custom date and time in source field

woodcock — Mon, 04 Feb 2019 20:23:48 GMT

You also have this wrong:

DATETIME_CONFIG = F:\Splunk\etc\apps\search\default\datetime.xml

It should be this:

DATETIME_CONFIG = \etc\apps\search\default\datetime.xml

Re: edit datetime.xml for my custom date and time in source field

ips_mandar — Tue, 05 Feb 2019 06:22:24 GMT

thanks for comment @woodcock
Yes I have already changed DateTime_config to this -

DATETIME_CONFIG = /etc/system/local/datetime.xml

and placed datetime.xml file in system/local path.
but still unable to parse timestamp from source file.

Re: edit datetime.xml for my custom date and time in source field

woodcock — Tue, 05 Feb 2019 14:45:55 GMT

That is the wrong place for it and has changed the way that Splunk handles unknown events. It should NOT go there (at least not with that name). Did you try putting it where I told you to put it and using th exact props setting that I showed you?

Re: edit datetime.xml for my custom date and time in source field

woodcock — Fri, 08 Feb 2019 15:17:09 GMT

If your distributing to indexers via a Cluster Master, use this:

\etc\slave-apps\search\default\datetime.xml

See here:
https://answers.splunk.com/answers/526680/splunk-ise-ta-fails-when-distributed-via-cluster-m.html

Re: edit datetime.xml for my custom date and time in source field

ips_mandar — Fri, 08 Feb 2019 15:25:58 GMT

Thanks for comment @woodcock
But I am not using cluster environment and using distributed environment and set props.conf in Heavy forwarder

Re: edit datetime.xml for my custom date and time in source field

woodcock — Fri, 08 Feb 2019 15:36:35 GMT

Check the error logs for datetime.xml for a hint.