Dynamic Sourcetype Extraction

[mattcg]

We're trying to set up a dynamic sourcetype extraction at index time. The reason for this is that we have about 40-50 different sourcetypes that would be generated in this format and we don't want to have 40-50 separate stanzas with a specific regex for each.

Is there a way to extract and set sourcetype dynamically?

Example Event:

<190>Sep 29 19:38:46 hostIP.ec2.internal INFO-ct-UserTransaction: userID="123456789" transactionType="WorkerAssign" itemID="156" taskType="WorkBay"...

transactionType will occur once at an inconsistent position in the payload after INFO-ct-UserTransaction:

WorkerAssign is representative of the value we want to extract as the sourcetype. We may have up to about 50 different values in its place.

What is the best way (if any) to extract and assign this sourcetype at index time?

[southeringtonp · Accepted solution]

This is very similar to:
http://answers.splunk.com/questions/6623/conditional-index-and-sourcetype-name-inputs-conf-by-file-n...

You just need to define a regex that will capture that value, and apply a transform to incoming events for that source or host. Something like:

# transforms.conf
[override-sourcetype]
SOURCE_KEY = source
DEST_KEY = MetaData:Sourcetype
REGEX = transactionType=\"([^\"]+)\"
FORMAT = sourcetype::$1

# props.conf
[source::/var/log/inputdir/*]
TRANSFORMS-sourcetype = override-sourcetype

For more information, take a look at:
http://www.splunk.com/base/Documentation/latest/Data/Advancedsourcetypeoverrides

[mattcg]

Strange I didn't see that question when I was searching. Thank you for the response, I'll give this a try.