topic source/sourcetype defined by folder names in Getting Data In

source/sourcetype defined by folder names

mhorn — Fri, 12 Apr 2013 18:34:01 GMT

is it possible to define the source and sourcetype fields to match a folder name? On each server our log structure for our products are as follows F:\Logs\Company_Name\Product\file.txt.

I'd like for the Company_Name folders to be defined as the source, and product folder to be defined as source type.

how do I go about doing this? I've read that this can be done in props for actual files, but I don't see an example for an actual folder location.

Re: source/sourcetype defined by folder names

Kate_Lawrence-G — Fri, 12 Apr 2013 20:17:14 GMT

It's actually done with both the props.conf and the transforms.conf

basically you have a props.conf kinda like this:

[source::F:\\Logs\\Company_Name\\Productfile.txt]
TRANSFORMS-setSourceType=setCompanySourceType

then a transforms.conf like this:

[setCompanySourceType]
SOURCE_KEY = MetaData:Source
REGEX = F:\\Logs\\(\w+_\w+)\\
FORMAT = sourcetype::$1
DEST_KEY = MetaData:Sourcetype

Now you can switch this around for whatever combination you need but basically you are assigning metaData thru regex.

Re: source/sourcetype defined by folder names

kristian_kolb — Fri, 12 Apr 2013 20:30:26 GMT

I guess you could do it off the source value, along these lines (have not tried it myself)

props.conf

[source::f:\Logs\...\*.txt]
TRANSFORMS-change_stuff = change_sourcetype, change_source

transforms.conf

[change_sourcetype]
SOURCE_KEY = MetaData:Source
REGEX = F:\\Logs\\[^\\]+\\([^\\]+)\\
DEST_KEY = MetaData:Sourcetype
FORMAT = sourcetype::$1

[change_source]
SOURCE_KEY = MetaData:Source
REGEX = F:\\Logs\\([^\\]+)\\
DEST_KEY = MetaData:Source
FORMAT = source::$1

However I think you should think a bit on whether you really want to do that. See this section in the manual;

http://docs.splunk.com/Documentation/Splunk/latest/Data/Whysourcetypesmatter

/Kristian

Re: source/sourcetype defined by folder names

kristian_kolb — Fri, 12 Apr 2013 20:33:42 GMT

Ooops, there I go for spending too much time editing. Should not watch movies while answering questions... 🙂

Re: source/sourcetype defined by folder names

mhorn — Fri, 12 Apr 2013 21:12:36 GMT

thanks for the reply guys. i'm going to work on this over the weekend and see how it goes.

Re: source/sourcetype defined by folder names

Ayn — Fri, 12 Apr 2013 22:14:27 GMT

That totally depends on which movie it is.

Re: source/sourcetype defined by folder names

mhorn — Mon, 15 Apr 2013 17:28:09 GMT

ok, I think I'm missing something as it appears to be pulling the data in as before. When I open the props/transform file from the system\default folder it states not to update that file, changes should be made in the system\local directory. So I copied both files and put them in the system\local directory and updated them as you suggested. I stopped the server and forwarder, then cleaned out the data and turned them on to pull it in again. Data appears to be coming in as before. By this I mean I would like to go to the search field and just type source="company_name". What am I missing?

Re: source/sourcetype defined by folder names

kristian_kolb — Mon, 15 Apr 2013 17:50:30 GMT

Hmm.. either there is something wrong with your props.conf stanza header, i.e. it does not match your logs (and thus do not being passed to the transform), or you are looking at old events (this only affects new data coming in).

Or there is a spelling error somewhere.

And as always, put them in $SPLUNK_HOME/etc/system/local for now. move to an app later if you want/need.

Re: source/sourcetype defined by folder names

mhorn — Mon, 15 Apr 2013 18:08:02 GMT

where exactly is the stanza header? maybe I don't have it in the correct location in the file. I didn't see in the props file a stanza header area.

Re: source/sourcetype defined by folder names

kristian_kolb — Mon, 15 Apr 2013 18:18:01 GMT

sorry bout the confusion, what I mean is where you put [source::blah blah]

perhaps you could post your props.conf and transforms.conf (relevant sections only). Anonymize as needed.

Re: source/sourcetype defined by folder names

mhorn — Mon, 15 Apr 2013 18:28:44 GMT

no need to apologize...still trying to figure out the splunk stuff, so i'm not making it easy on you!

Re: source/sourcetype defined by folder names

mhorn — Mon, 15 Apr 2013 19:38:48 GMT

now I'm updating the props/transform on my splunk server, not on the forwarder correct?
I'd attach the two files, but I don't see a button to do this, and copy/paste surpasses the number of characters allowed in the boxes.

Re: source/sourcetype defined by folder names

kristian_kolb — Mon, 15 Apr 2013 20:01:08 GMT

on the server, yes. Unless it's a heavy forwarder.
No you can't attach files like that, but you could just edit your original post with the relevant portions of the props.conf and transforms.conf files