topic Re: How to edit my datetime.xml to extract the date from a filename to use as the date for indexed events? in Getting Data In

How to edit my datetime.xml to extract the date from a filename to use as the date for indexed events?

moaf13 — Thu, 03 Nov 2016 17:12:27 GMT

I have read multiple blogs and answers and couldn't find anything that helps.

The filename will be something like thisblah_blah_blah....161030.txt
I have checked my regex on regex101 and it works perfectly.

This is what I have,

props.conf

[filedate]
DATETIME_CONFIG = /etc/system/local/datetime.xml
category= Date_time
CHARSET=AUTO
disabled=false
pulldown_type = true

Datetime.xml

<datetime>
  <define name="_masheddate3" extract="year, month, day,">
        <text> ![CDATA[(?:^|source:|source::).*?([123]\d)([01]\d)([0123]\d)[^0-9]]]  </text>
  </define>
  <datePatterns>
        <use name="_masheddate3"/>
  </datePatterns>
</datetime>

And this the error message that i get from Splunkd.log

11-03-2016 12:44:36.860 -0400 ERROR AggregatorMiningProcessor - Uncaught exception in Aggregator, skipping an event: Error parsing regex XML file: C:\Program Files\Splunk\etc\system\local\datetime.xml - Couldn't find 'timePatterns' in config data for AggregatorProcessor. - data_source="C:\Users\ComputerName\Documents\Test2\blah_blah161030.txt", data_host="local", data_sourcetype="filedate"</code>

11-03-2016 12:44:36.860 -0400 ERROR AggregatorMiningProcessor - Uncaught exception in Aggregator, skipping an event: Error parsing regex XML file: C:\Program Files\Splunk\etc\system\local\datetime.xml - Couldn't find 'timePatterns' in config data for AggregatorProcessor. - data_source="C:\Users\ComputerName\Documents\Test2\blah_blah161030.txt", data_host="local", data_sourcetype="filedate"

I know Props.conf is working by looking at this error message, so the only issue is in datetime.xml...
The time is irrelevant to me. How do I make Splunk just get the date and ignore the time? If there is anyone that had success in this, please let me know!

Re: How to edit my datetime.xml to extract the date from a filename to use as the date for indexed events?

moaf13 — Thu, 03 Nov 2016 22:10:17 GMT

Update: I was able to fix that error or by pass it by adding,

However, when that happens, Splunk automatically sets timestamp as none
which basically forces it to index it at a current time.

Re: How to edit my datetime.xml to extract the date from a filename to use as the date for indexed events?

sshelly_splunk — Fri, 04 Nov 2016 15:25:33 GMT

I would not mess with datetime, unless really required. This should be able to set your day, month year (I think:)) :

EXTRACT-time = (?\d{8}) in source
EVAL-_time = strptime(time,"%Y%m%d”)

It has been a while since i used this, but it might help. Note that the time of day for each event in that file will be midnight.

Re: How to edit my datetime.xml to extract the date from a filename to use as the date for indexed events?

kjetilho — Thu, 17 Nov 2016 09:26:40 GMT

guess you would need masheddate3 and not masheddate2

Re: How to edit my datetime.xml to extract the date from a filename to use as the date for indexed events?

Lowell — Thu, 02 Feb 2017 00:56:54 GMT

Um, doesn't setting this at search time have a pretty significant performance implication? Additionally, since data is stored in reverse TIME order (emphasis on time) in the index, this means that what your search pull back will be based on the initial _time field (as it was stored in the index) and NOT search-time extracted _time field shown above.

I strongly recommend avoiding this solution. It's find if you want to extract it with some OTHER name, just don't use "_time".

Re: How to edit my datetime.xml to extract the date from a filename to use as the date for indexed events?

abhishekdharga — Fri, 03 Nov 2017 20:56:07 GMT

Thanks it worked

Re: How to edit my datetime.xml to extract the date from a filename to use as the date for indexed events?

maraman_splunk — Sun, 05 Nov 2017 12:59:06 GMT

This config is search time only
Time should be set at index time.

Re: How to edit my datetime.xml to extract the date from a filename to use as the date for indexed events?

maraman_splunk — Sun, 05 Nov 2017 13:13:32 GMT

I think Splunk complain because you don't specify any time in your datetime.xml
so you should add something that just set a time to a known value (perhaps at 00:00:00)

   1 <define name="Your Custom Name Time" >
   2          <text>set your time to 00:00:00 here</text>
   3  </define>
   4  
   5  <timePatterns>
   6       <use name="Your Custom Name Time"/>
   7 </timePatterns>