topic Re: dynamic sourcetypes - can splunk do this? (and I'll be impressed if it can) in Dashboards & Visualizations

dynamic sourcetypes - can splunk do this? (and I'll be impressed if it can)

a212830 — Fri, 06 Feb 2015 14:11:04 GMT

Hi,

I have a request to monitor a directory, with dynamic logfiles. Sometimes they are there, sometimes the customer will create new ones.... They have validated that the logfiles all follow the same format. Is it possible for splunk to create a sourcetype based upon the name of the file? They all follow the format SERVICE_PID.log. I could create one based upon the directory name, but they would rather use separate sourcetypes, if possible, for easier analysis and reporting.

Re: dynamic sourcetypes - can splunk do this? (and I'll be impressed if it can)

MuS — Fri, 06 Feb 2015 14:15:53 GMT

Hi a212830,

Yes it can 😉 You will need to specify this in your props/transforms files any where parsing/indexing is being performed.

props.conf

[source::...regex_to_match_filename] 
TRANSFORMS-fs = force-sourcetype-st

transforms.conf

[force-sourcetype-st] 
DEST_KEY = MetaData::Sourcetype 
SOURCE_KEY = MetaData::Source 
REGEX = YOUR_REGEX_TO_PULL_THE_FILENAME 
FORMAT = sourcetype::$1 
WRITE_META = true

Hope this helps ...

cheers, MuS

Re: dynamic sourcetypes - can splunk do this? (and I'll be impressed if it can)

thomrs — Fri, 06 Feb 2015 14:18:39 GMT

Source type is set at index time so do not think this is possible. I had a similar issue and I used the same source type for everything but added a new filed based on the file source called 'logname'. This is a search time approach and has been working fine.

You could use a transform at index time to add the 'logname' as metadata if needed.

Re: dynamic sourcetypes - can splunk do this? (and I'll be impressed if it can)

a212830 — Fri, 06 Feb 2015 16:34:55 GMT

Good stuff, both useful. I'll try them out.

Re: dynamic sourcetypes - can splunk do this? (and I'll be impressed if it can)

thomrs — Fri, 06 Feb 2015 17:17:21 GMT

I was wrong you can do this with an index time transform. I always stay away from doing that, i never want to change the raw data,

http://docs.splunk.com/Documentation/Splunk/latest/Data/Advancedsourcetypeoverrides

Re: dynamic sourcetypes - can splunk do this? (and I'll be impressed if it can)

MuS — Fri, 06 Feb 2015 19:30:13 GMT

Good one! Knowing his weakness, is a way to strength .... or something like that ..... Yoda would said have 🙂