datetime.xml filename timestamp !?!

[jamesbaud]

Hello,

I am trying to extract the timestamp from the filepath of my log files. I've read and followed variations of what was mentioned in the below posts, but have not successfully been able to extract the timestamp from the the filepath.

http://answers.splunk.com/questions/3055/creating-a-masheddate3-in-datetime-xml http://answers.splunk.com/questions/5960/extract-a-field-from-event-source-filename

Here is what I have.

filepath /Users/guest/stuff/AFR/2010-09-06_0602_FOOFOO_BLAH_FILES/foo/somefile.log

date.xml

<datetime>


<define name="_masheddate3" extract="year, month, day, hour, minute">
        <text><![CDATA[source::.*?/AFR/(20\d\d)-(0\d|1[012])-([012]\d|3[01])[_](\d\d)(\d\d)[_].*?[_].*?[_]FILES/.*$]]></text>
</define>


<timePatterns>
      <use name="_masheddate3"/>
</timePatterns>
<datePatterns>
      <use name="_masheddate3"/>
</datePatterns>

</datetime>

I've tried adding the _masheddate3 to a copy of the existing datetime.xml, but still no difference. What I copied above is the full content of my "copy" of datetime.xml -- I want all my events to get their timestamp from the filepath.

props.conf

[host::*]
DATETIME_CONFIG = /etc/system/local/datetime.xml

I've also tried

[<sourcetype>]
DATETIME_CONFIG = /etc/system/local/datetime.xml
<other line breaking attributes>

when I defined DATETIME_CONFIG with a sourcetype stanza, it completed messed up all my linebreaking logic. When I use it on hostname, it did not mess up my line breaking logic.

debug output from splunkd.log

09-09-2010 00:46:30.794 INFO  DateParserVerbose - Setting maxDaysAgo=1825 and maxDaysHence=7
09-09-2010 00:46:30.794 DEBUG LoadDateParserRegexes - put _masheddate3 regex=source::.*?/AFR/(20\d\d)-(0\d|1[012])-([012]\d|3[01])[_](\d\d)(\d\d)[_].*?[_].*?[_]FILES/.*$
09-09-2010 00:46:30.795 DEBUG LoadDateParserRegexes -     * year
09-09-2010 00:46:30.795 DEBUG LoadDateParserRegexes -     * month
09-09-2010 00:46:30.795 DEBUG LoadDateParserRegexes -     * day
09-09-2010 00:46:30.795 DEBUG LoadDateParserRegexes -     * hour
09-09-2010 00:46:30.795 DEBUG LoadDateParserRegexes -     * minute
09-09-2010 00:46:30.796 INFO  DateParserVerbose - Setting maxDaysAgo=2000 and maxDaysHence=2
09-09-2010 00:46:33.791 INFO  DateParserVerbose - Setting maxDaysAgo=1825 and maxDaysHence=7
09-09-2010 00:46:33.792 INFO  DateParserVerbose - Setting maxDaysAgo=2000 and maxDaysHence=2

why isn't the timestamp getting parsed out correctly from the filepath?

[meno]

A bit difficult to answer...

Timestamp extraction follows these rules: http://www.splunk.com/base/Documentation/latest/Admin/HowSplunkextractstimestamps

I would not touch datetime.xml as it is a very good base for Splunk's automatic timestamp recognition from the event itself.

If your events do not have any timestamps could you post here what exactly is your result in Splunk without manipulating datetime.xml ? Probably the file modification time works already fine for you ?

If your events have timestamps but recognition is bad, an example would also help here.

[jamesbaud]

most of the sources do NOT have a timestamp within the event, but some do. In any case, I need to extract the timestamp from the filename. From a couple other posts I've read, it seems like this can be done by what I've done above. However, I am not sure why in my situation, it does not extract the date/time from the filename.