topic Re: How i set date by log path address ? in Splunk Search

How i set date by log path address ?

nitesh218ss — Tue, 12 May 2015 06:48:44 GMT

Hi have a log which is inside folder which folder name is date
i give folder name or path is=
C:\Users\T_NiteshS1\Documents\My Received Files\20150511\log2.log

If you see before log2.log you get folder 20150511 This is date
if you expend 20150511 this yyyymmdd

how is set in datetime.xml i don't now xml

Re: How i set date by log path address ?

nitesh218ss — Tue, 12 May 2015 07:47:22 GMT

i use in xml

&lt;define name="_masheddate2" extract="month, day, year"&gt;
    &lt;text&gt;&lt;![CDATA[(?:^|C:\Program Files\Splunk\etc\apps\search::).*?(20\d{2})(\d{2})(\d{2})]]&gt;&lt;/text&gt;
&lt;/define&gt;

i use in props.conf
DATETIME_CONFIG = C:\Program Files\Splunk\etc\apps\search\datetime.xml

Re: How i set date by log path address ?

woodcock — Tue, 12 May 2015 16:34:24 GMT

Try this for your datetime.xml (hopefully markdown will not mangle the text):

<datetime>
<define name="_dateFromDirectorySegment" extract="year, month, day">
<text><![CDATA[source::.*?(\d{4})(\d{2})(\d{2})]]></text>
</define>
<define name="_timeFromEventData" extract="hour, minute, second, subsecond">
<text><![CDATA[^(\d{2}):(\d{2}:(\d{2}.(\d{3}]]></text>
</define>
<timePatterns>
<use name="_timeFromEventData"/>
</timePatterns>
<datePatterns>
<use name="_dateFromDirectorySegment"/>
</datePatterns>
</datetime>

Re: How i set date by log path address ?

nitesh218ss — Tue, 12 May 2015 19:11:43 GMT

in this you not given file path any where

C:\Users\T_NiteshS1\Documents\My Received Files\20150511\log2.log

This is my file path
with out given file path how they take date?

Today i try with file path address that y they not take date
ok thanks i try this in office tomorrow

Re: How i set date by log path address ?

woodcock — Tue, 12 May 2015 20:12:51 GMT

Yes, I skip over the path because you might change the segment-level (add/remove path segments) so I left is as wildcarded. This configuration WILL work if you try it.

Re: How i set date by log path address ?

nitesh218ss — Wed, 13 May 2015 08:18:04 GMT

I try your answer but they not work after i try many way but fail
itry

[CDATA[source::.*?\\[My]+\ [Received]+\ [Files]+\\(\d{4})(\d{2})(\d{2})\\[NB92\-Transaction07.log]+]

[CDATA[source::.*?\\(\d{4})(\d{2})(\d{2})\\[NB92\-Transaction07.log]+]

[CDATA[source::.*?\\(\d{4})(\d{2})(\d{2})\\]

[CDATA[source::.*?\\(\d{4})(\d{2})(\d{2})]

[CDATA[source::.*?\My Received Files\(\d{4})(\d{2})(\d{2})]

and so many way i try

logfile not contain date logfile present inside folder that folder have date as name

Re: How i set date by log path address ?

woodcock — Wed, 13 May 2015 14:06:24 GMT

The problem is your entry in props.conf; it uses a relative path and you have given it a full path. Use my original datetime.xml file and use this in props.conf:
DATETIME_CONFIG = /etc/apps/search/datetime.xml

Re: How i set date by log path address ?

nitesh218ss — Wed, 13 May 2015 17:24:43 GMT

i used both full address and /etc/apps/search/datetime.xml but they not work

but when i put datetime.xml in /etc/apps/search/local/datetime.xml then at indexing time
splunk not read file give message check sourcetype setting

when i used /etc/apps/search/datetime.xml they read at indexing time normally

Re: How i set date by log path address ?

nitesh218ss — Fri, 15 May 2015 09:34:16 GMT

when i put date in file name like tran20150511.log
and in datetime.xml seeting for date part i useis: [CDATA[source::.*?tran(\d{4})(\d{2})(\d{2})]]
and in props.conf

DATETIME_CONFIG = C:\Program Files\Splunk\etc\apps\search\datetimee.xml
EXTRACT-fields = (?&lt;time&gt;\d{2}:\d{2}:\d{2}\.\d{3})\|(?&lt;audit&gt;[^\|]+)\|(?&lt;serviceType&gt;[^\|]+)\|(?&lt;processId&gt;[^\|]+)\|(?&lt;serviceName&gt;[^\|]+)\|(?&lt;indicator&gt;[^\|]+)\|(?P&lt;message&gt;
LINE_BREAKER = ([\r\n]+)\d{2}:\d{2}:\d{2}\.\d{3}\|[^\|]+\|[^\|]+\|[^\|]+\|[^\|]+\|[^\|]+\|
NO_BINARY_CHECK = true
SHOULD_LINEMERGE = false
category = Custom
disabled = false
pulldown_type = true

Then they take date correctly but

by folder they not take date

Re: How i set date by log path address ?

woodcock — Fri, 15 May 2015 15:39:08 GMT

So you have datetime.xml working generally but we cannot get the exat text string to do it by path. That is a good start. To summarize, your desired path is like this:

C:\Users\T_NiteshS1\Documents\My Received Files\20150511\log2.log

So this line should work (do not worry that it is not fully qualified):

<define name="_dateFromDirectorySegment" extract="year, month, day">
<text><![CDATA[source::.*?(\d{4})(\d{2})(\d{2})]]></text>

Yes, this is the same thing that I wrote way back in the beginning but I stand by my position that it should work (unless you have copied it by hand and typoed something).

Do note that the first line is very important because it specifies the order of the captured strings and how they apply to the variables.

Re: How i set date by log path address ?

nitesh218ss — Sat, 16 May 2015 16:39:00 GMT

sir i tired this so many time but they not worked
i thing some thing is wrong in splunk to take date from folder name
this is bug in spluk possible

Re: How i set date by log path address ?

woodcock — Fri, 07 Aug 2015 00:10:35 GMT

Maybe because you are windows, the problem is the path so instead of this:

DATETIME_CONFIG = /etc/apps/search/datetime.xml

Try this:

DATETIME_CONFIG = \etc\apps\search\datetime.xml