topic Re: assign sourcetype based upon file name in Getting Data In

assign sourcetype based upon file name

a212830 — Mon, 11 Mar 2013 12:58:30 GMT

Hi,

Is there a way to dynamically assign sourcetype based upon input filename?

Re: assign sourcetype based upon file name

alacercogitatus — Mon, 11 Mar 2013 13:12:52 GMT

There is. You will need to specify this in your props/transforms files any where indexing is being performed.

props.conf [source::...regex_to_match_filename] TRANSFORMS-fs = force-sourcetype-st

transforms.conf [force-sourcetype-st] DEST_KEY = MetaData::Sourcetype SOURCE_KEY = MetaData::Source REGEX = YOUR_REGEX_TO_PULL_THE_FILENAME FORMAT = sourcetype::$1 WRITE_META = true

Referances
http://docs.splunk.com/Documentation/Splunk/latest/admin/Transformsconf
http://docs.splunk.com/Documentation/Splunk/latest/admin/Propsconf

Re: assign sourcetype based upon file name

jrodman — Mon, 28 Sep 2020 13:29:48 GMT

alacercogitatus's answer is certainly correct. You can set the sourcetype at parsing time (typically on your indexer) with props and transforms. For some problems this is the way to go.

However, if you want the sourcetype set up from the beginning, so you can control things like the "CHARSET" (encoding), NO_BINARY_CHECK and so on, then you can also control the sourcetype set at the input layer in tailing. For this to work, you need to be willing to put configuration on your forwarder, and it would look something like

props.conf:

[source::< pattern >]
sourcetype = what_you_want

< pattern > is whatever you need to match the file. The simplest thing that gets everything you want and doesn't get false positives is kind of a local decision, but often it looks something like:



    [source::/var/log/my_app/*.log]

    sourcetype = my_app

If you use both these techniques, then the value set at the input layer will be used until it is possibly modified by the props/transforms pass. Usually though, you want to use one approach or the other for a particular dataset, lest you make things too confusing.

Re: assign sourcetype based upon file name

mbonsack_splunk — Mon, 14 Sep 2015 23:36:28 GMT

alacercogitatus's answer does work, though the separators for the MetaData keys are single colons, not double, i.e.

DEST_KEY = MetaData:Sourcetype
SOURCE_KEY = MetaData:Source

After that it worked great!