Splunk Search

Dynamic Sourcetype Extraction

Explorer

We're trying to set up a dynamic sourcetype extraction at index time. The reason for this is that we have about 40-50 different sourcetypes that would be generated in this format and we don't want to have 40-50 separate stanzas with a specific regex for each.

Is there a way to extract and set sourcetype dynamically?

Example Event:

<190>Sep 29 19:38:46 hostIP.ec2.internal INFO-ct-UserTransaction: userID="123456789" transactionType="WorkerAssign" itemID="156" taskType="WorkBay"...

transactionType will occur once at an inconsistent position in the payload after INFO-ct-UserTransaction:

WorkerAssign is representative of the value we want to extract as the sourcetype. We may have up to about 50 different values in its place.

What is the best way (if any) to extract and assign this sourcetype at index time?

1 Solution

Motivator

This is very similar to:
http://answers.splunk.com/questions/6623/conditional-index-and-sourcetype-name-inputs-conf-by-file-n...

You just need to define a regex that will capture that value, and apply a transform to incoming events for that source or host. Something like:

# transforms.conf
[override-sourcetype]
SOURCE_KEY = source
DEST_KEY = MetaData:Sourcetype
REGEX = transactionType=\"([^\"]+)\"
FORMAT = sourcetype::$1

# props.conf
[source::/var/log/inputdir/*]
TRANSFORMS-sourcetype = override-sourcetype

For more information, take a look at:
http://www.splunk.com/base/Documentation/latest/Data/Advancedsourcetypeoverrides

View solution in original post

Motivator

This is very similar to:
http://answers.splunk.com/questions/6623/conditional-index-and-sourcetype-name-inputs-conf-by-file-n...

You just need to define a regex that will capture that value, and apply a transform to incoming events for that source or host. Something like:

# transforms.conf
[override-sourcetype]
SOURCE_KEY = source
DEST_KEY = MetaData:Sourcetype
REGEX = transactionType=\"([^\"]+)\"
FORMAT = sourcetype::$1

# props.conf
[source::/var/log/inputdir/*]
TRANSFORMS-sourcetype = override-sourcetype

For more information, take a look at:
http://www.splunk.com/base/Documentation/latest/Data/Advancedsourcetypeoverrides

View solution in original post

Explorer

Strange I didn't see that question when I was searching. Thank you for the response, I'll give this a try.

0 Karma
State of Splunk Careers

Access the Splunk Careers Report to see real data that shows how Splunk mastery increases your value and job satisfaction.

Find out what your skills are worth!