topic Re: creating a _masheddate3 in datetime.xml in Splunk Search

creating a _masheddate3 in datetime.xml

hiddenkirby — Thu, 27 May 2010 23:54:48 GMT

This is a question stemmed from http://answers.splunk.com/questions/2996/strptime-format-for-yyyymmddhhmmss and http://answers.splunk.com/questions/2831/index-on-regex-field-from-source

my event source looks like this ".../scripts/201005271243/data/file.txt".

End goal is to parse the date from the source.

I have copied the datetime.xml file to myapp/default dir.

i have modified props.conf to

[my_sourcetype]
DATETIME_CONFIG = /opt/splunk/etc/apps/myapp/default/datetime.xml

i have modified datetime.xml to

<define name="_masheddate3" extract="year, month, day, hour, minute">
    <text><![CDATA[(?:^|source::).*\/scripts\/(\d{4})(\d{2})(\d{2})(\d{2})(\d{2})\/data.*]]></text>
</define>

as well as the

<datePatterns><use name="_masheddate3"/>

...to no avail. Index doesn't seem to populate at all.

Re: creating a _masheddate3 in datetime.xml

hiddenkirby — Fri, 28 May 2010 01:03:33 GMT

i suspect the regex to be incorrect.

Re: creating a _masheddate3 in datetime.xml

Lowell — Fri, 28 May 2010 03:07:30 GMT

Well, your regex does seem to work, however, you may want to tweak it to (1) only match source:: patterns, although you have enough other path-like matching it's unlikely to match else where, and (2) remove the unnecessary escaping for /, but I'm not sure that would cause you a problem.

I'd suggest, something like this:

<text><![CDATA[source::.*?/scripts/(\d{4})(\d{2})(\d{2})(\d{2})(\d{2})/data/]]></text>

I also dropped off the .* from the end since the other rules don't have any such matching at the end. I also replaced your .* at the front with the non-greedy .*? which, should help performance.

Hmmm, I think this could be your problem.... Your _masheddate3 is contains not only a date but also a time. It looks like _combdatetime for example, is setup in both the timePatterns listing as well as the datePatterns. So if you don't have your entry in both, then I'd give that a try.

Re: creating a _masheddate3 in datetime.xml

hiddenkirby — Fri, 28 May 2010 03:50:05 GMT

how do i know if it failed.. other than finding the index did not populate? is an error in _internal ?

Re: creating a _masheddate3 in datetime.xml

hiddenkirby — Fri, 28 May 2010 03:55:55 GMT

Ah hah! "Can't open DatePaser XML configuration file ...datetime.xml" No such file or directory.

Re: creating a _masheddate3 in datetime.xml

hiddenkirby — Fri, 28 May 2010 04:09:53 GMT

bad path to datetime.xml

Re: creating a _masheddate3 in datetime.xml

Lowell — Fri, 28 May 2010 04:29:54 GMT

lol. Sometimes it's the simple things that take the most time to find. (I've been there many many times). Good catch. Hey, let me know about the whole time/date Patterns thing, does it make a difference?

Re: creating a _masheddate3 in datetime.xml

gkanapathy — Fri, 28 May 2010 06:44:26 GMT

You do need to list the name of your rule in both the datePatterns and timePatterns part of the datetime.xml if you want it to get both date and time. It's okay to list the same rule in both places.

Re: creating a _masheddate3 in datetime.xml

hiddenkirby — Fri, 28 May 2010 23:46:32 GMT

I added it to both time and date .. still no go. Seems to still run off of the last modified date. New question... http://answers.splunk.com/questions/3102/datetime-xml-change-doesnt-seem-to-be-working

Re: creating a _masheddate3 in datetime.xml

nitesh218ss — Wed, 13 May 2015 09:30:09 GMT

Hi have a log which is inside folder which folder name is date
i give folder name or path is=
C:\Users\T_NiteshS1\Documents\My Received Files\20150511\log2.log

If you see before log2.log you get folder 20150511 This is date
if you expend 20150511 this yyyymmdd

how is set in datetime.xml i don't now xml

I try many way but fail
my try
[CDATA[source::.*?\[My]+\ [Received]+\ [Files]+\(\d{4})(\d{2})(\d{2})\[NB92-Transaction07.log]+]

[CDATA[source::.*?\(\d{4})(\d{2})(\d{2})\[NB92-Transaction07.log]+]

[CDATA[source::.*?\(\d{4})(\d{2})(\d{2})\]

[CDATA[source::.*?\(\d{4})(\d{2})(\d{2})]

[CDATA[source::.*?\My Received Files(\d{4})(\d{2})(\d{2})]

and so many way i try