Splunk Search

Dynamic Sourcetype Extraction


We're trying to set up a dynamic sourcetype extraction at index time. The reason for this is that we have about 40-50 different sourcetypes that would be generated in this format and we don't want to have 40-50 separate stanzas with a specific regex for each.

Is there a way to extract and set sourcetype dynamically?

Example Event:

<190>Sep 29 19:38:46 hostIP.ec2.internal INFO-ct-UserTransaction: userID="123456789" transactionType="WorkerAssign" itemID="156" taskType="WorkBay"...

transactionType will occur once at an inconsistent position in the payload after INFO-ct-UserTransaction:

WorkerAssign is representative of the value we want to extract as the sourcetype. We may have up to about 50 different values in its place.

What is the best way (if any) to extract and assign this sourcetype at index time?


Re: Dynamic Sourcetype Extraction


This is very similar to:

You just need to define a regex that will capture that value, and apply a transform to incoming events for that source or host. Something like:

# transforms.conf
SOURCE_KEY = source
DEST_KEY = MetaData:Sourcetype
REGEX = transactionType=\"([^\"]+)\"
FORMAT = sourcetype::$1

# props.conf
TRANSFORMS-sourcetype = override-sourcetype

For more information, take a look at:

View solution in original post


Re: Dynamic Sourcetype Extraction


Strange I didn't see that question when I was searching. Thank you for the response, I'll give this a try.

0 Karma