How to change sourcetype on indexer based on the value of source?

[deepak02]

Hi,

My data flows in from the forwarder where index=idx1 and sourcetype=sourcetypeA have been set using inputs.conf.

On the indexer, I want the following rule to be set for the same data, (i.e. I want to change the sourcetype based on the value of source)

If source=/abc/logs/server-*-error.log, sourcetype=sourcetypeBError
If source=/abc/logs/server-*-transaction.log, sourcetype=sourcetypeBTransaction
If source=/abc/logs/server-*-access.log, sourcetype=sourcetypeBAccess

How do I set up this rule on the indexer?
I am using Splunk Enterprise.

Thanks,
Deepak

[kmorris_splunk]

Try this:

Provide an initial sourcetype for your data in your inputs.conf. This will remain the sourcetype if none of your transforms stanzas match. In your props.conf you point at the different stanza names in your transforms.conf file (set_sourcetype_1 and set_sourcetype_2 in my example).

In your props.conf:

[mysourcetype]
TRANSFORMS-set_sourcetype = set_sourcetype_1, set_sourcetype_2

In your transforms.conf:

[set_sourcetype_1]
SOURCE_KEY = MetaData:Source
REGEX = ^source::/path/to/the/file/source1.txt
FORMAT = sourcetype::sourcetype_1
DEST_KEY = MetaData:Sourcetype

[set_sourcetype_2]
SOURCE_KEY = MetaData:Source
REGEX = ^source::/path/to/the/file/source2.txt
FORMAT = sourcetype::sourcetype_2
DEST_KEY = MetaData:Sourcetype

In this example, you are routing it to the appropriate transforms.conf stanzas based on the original sourcetype. Each transforms.conf stanza looks at the source field and determines if the REGEX matches. If it does, it sets the sourcetype to the value after the :: in the FORMAT = sourcetype:: line.

[kmorris_splunk]

To adonio's point, this could be done in the inputs.conf.

[deepak02]

Thanks for the response.

Unfortunately my forwarders are owned by a different team, and they are not willing to change anything for now. I will try the above settings and see if it works.

[kmorris_splunk]

Ok. If that is the case, then you could use the props.conf and transforms.conf method I describe, however, it would probably be more efficient doing it at the Universal Forwarder side if you can get them to make the change in the future.

[adonio]

why not on the forwarder?
inputs.conf

[monitor:///abc/logs/server-*-error.log]
index = idx1
sourcetype=sourcetypeBError

[monitor:///abc/logs/server-*-transaction.log]
index = idx1
sourcetype=sourcetypeBTransaction

[monitor:///abc/logs/server-*-access.log]
index = idx1
sourcetype=sourcetypeBAccess

if you insist setting it up on the indexer
there are many answers here on this subject, here are couple examples:
https://answers.splunk.com/answers/410924/dynamic-sourcetype-based-on-source-not-working.html
https://answers.splunk.com/answers/214598/dynamic-sourcetypes-can-splunk-do-this-and-ill-be.html
https://answers.splunk.com/answers/368330/dynamically-assign-sourcetype-on-folder.html

hope it helps

[deepak02]

Thankyou.

Unfortunately my forwarders are owned by a different team, and they are not willing to change anything for now.

I will follow your advice.

[adonio]

check the links in my answer or follow kmorris answer.
good luck