Getting Data In

How to change sourcetype on indexer based on the value of source?

Path Finder

Hi,

My data flows in from the forwarder where index=idx1 and sourcetype=sourcetypeA have been set using inputs.conf.

On the indexer, I want the following rule to be set for the same data, (i.e. I want to change the sourcetype based on the value of source)

If source=/abc/logs/server-*-error.log, sourcetype=sourcetypeBError
If source=/abc/logs/server-*-transaction.log, sourcetype=sourcetypeBTransaction
If source=/abc/logs/server-*-access.log, sourcetype=sourcetypeBAccess

How do I set up this rule on the indexer?
I am using Splunk Enterprise.

Thanks,
Deepak

0 Karma

Splunk Employee
Splunk Employee

Try this:

Provide an initial sourcetype for your data in your inputs.conf. This will remain the sourcetype if none of your transforms stanzas match. In your props.conf you point at the different stanza names in your transforms.conf file (setsourcetype1 and setsourcetype2 in my example).

In your props.conf:

[mysourcetype]
TRANSFORMS-set_sourcetype = set_sourcetype_1, set_sourcetype_2

In your transforms.conf:

[set_sourcetype_1]
SOURCE_KEY = MetaData:Source
REGEX = ^source::/path/to/the/file/source1.txt
FORMAT = sourcetype::sourcetype_1
DEST_KEY = MetaData:Sourcetype

[set_sourcetype_2]
SOURCE_KEY = MetaData:Source
REGEX = ^source::/path/to/the/file/source2.txt
FORMAT = sourcetype::sourcetype_2
DEST_KEY = MetaData:Sourcetype

In this example, you are routing it to the appropriate transforms.conf stanzas based on the original sourcetype. Each transforms.conf stanza looks at the source field and determines if the REGEX matches. If it does, it sets the sourcetype to the value after the :: in the FORMAT = sourcetype:: line.

Splunk Employee
Splunk Employee

To adonio's point, this could be done in the inputs.conf.

0 Karma

Path Finder

Thanks for the response.

Unfortunately my forwarders are owned by a different team, and they are not willing to change anything for now. I will try the above settings and see if it works.

0 Karma

Splunk Employee
Splunk Employee

Ok. If that is the case, then you could use the props.conf and transforms.conf method I describe, however, it would probably be more efficient doing it at the Universal Forwarder side if you can get them to make the change in the future.

0 Karma

SplunkTrust
SplunkTrust

why not on the forwarder?
inputs.conf

[monitor:///abc/logs/server-*-error.log]
index = idx1
sourcetype=sourcetypeBError

[monitor:///abc/logs/server-*-transaction.log]
index = idx1
sourcetype=sourcetypeBTransaction

[monitor:///abc/logs/server-*-access.log]
index = idx1
sourcetype=sourcetypeBAccess

if you insist setting it up on the indexer
there are many answers here on this subject, here are couple examples:
https://answers.splunk.com/answers/410924/dynamic-sourcetype-based-on-source-not-working.html
https://answers.splunk.com/answers/214598/dynamic-sourcetypes-can-splunk-do-this-and-ill-be.html
https://answers.splunk.com/answers/368330/dynamically-assign-sourcetype-on-folder.html

hope it helps

Path Finder

Thankyou.

Unfortunately my forwarders are owned by a different team, and they are not willing to change anything for now.

I will follow your advice.

0 Karma

SplunkTrust
SplunkTrust

check the links in my answer or follow kmorris answer.
good luck