dynamic sourcetypes - can splunk do this? (and I'll be impressed if it can)

[a212830]

Hi,

I have a request to monitor a directory, with dynamic logfiles. Sometimes they are there, sometimes the customer will create new ones.... They have validated that the logfiles all follow the same format. Is it possible for splunk to create a sourcetype based upon the name of the file? They all follow the format SERVICE_PID.log. I could create one based upon the directory name, but they would rather use separate sourcetypes, if possible, for easier analysis and reporting.

[thomrs]

Source type is set at index time so do not think this is possible. I had a similar issue and I used the same source type for everything but added a new filed based on the file source called 'logname'. This is a search time approach and has been working fine.

You could use a transform at index time to add the 'logname' as metadata if needed.

[thomrs]

I was wrong you can do this with an index time transform. I always stay away from doing that, i never want to change the raw data,

http://docs.splunk.com/Documentation/Splunk/latest/Data/Advancedsourcetypeoverrides

[MuS]

Good one! Knowing his weakness, is a way to strength .... or something like that ..... Yoda would said have 🙂

[MuS · Accepted solution]

Hi a212830,

Yes it can 😉 You will need to specify this in your props/transforms files any where parsing/indexing is being performed.

props.conf

[source::...regex_to_match_filename] 
TRANSFORMS-fs = force-sourcetype-st

transforms.conf

[force-sourcetype-st] 
DEST_KEY = MetaData::Sourcetype 
SOURCE_KEY = MetaData::Source 
REGEX = YOUR_REGEX_TO_PULL_THE_FILENAME 
FORMAT = sourcetype::$1 
WRITE_META = true

Hope this helps ...

cheers, MuS

[a212830]

Good stuff, both useful. I'll try them out.