We're trying to set up a dynamic sourcetype extraction at index time. The reason for this is that we have about 40-50 different sourcetypes that would be generated in this format and we don't want to have 40-50 separate stanzas with a specific regex for each.
Is there a way to extract and set sourcetype dynamically?
<190>Sep 29 19:38:46 hostIP.ec2.internal INFO-ct-UserTransaction: userID="123456789" transactionType="WorkerAssign" itemID="156" taskType="WorkBay"...
transactionType will occur once at an inconsistent position in the payload after INFO-ct-UserTransaction:
WorkerAssign is representative of the value we want to extract as the sourcetype. We may have up to about 50 different values in its place.
What is the best way (if any) to extract and assign this sourcetype at index time?
You just need to define a regex that will capture that value, and apply a transform to incoming events for that source or host. Something like:
# transforms.conf [override-sourcetype] SOURCE_KEY = source DEST_KEY = MetaData:Sourcetype REGEX = transactionType=\"([^\"]+)\" FORMAT = sourcetype::$1 # props.conf [source::/var/log/inputdir/*] TRANSFORMS-sourcetype = override-sourcetype
For more information, take a look at: