Splunk Search

Dynamic Sourcetype Extraction

mattcg
Explorer

We're trying to set up a dynamic sourcetype extraction at index time. The reason for this is that we have about 40-50 different sourcetypes that would be generated in this format and we don't want to have 40-50 separate stanzas with a specific regex for each.

Is there a way to extract and set sourcetype dynamically?

Example Event:

<190>Sep 29 19:38:46 hostIP.ec2.internal INFO-ct-UserTransaction: userID="123456789" transactionType="WorkerAssign" itemID="156" taskType="WorkBay"...

transactionType will occur once at an inconsistent position in the payload after INFO-ct-UserTransaction:

WorkerAssign is representative of the value we want to extract as the sourcetype. We may have up to about 50 different values in its place.

What is the best way (if any) to extract and assign this sourcetype at index time?

1 Solution

southeringtonp
Motivator

This is very similar to:
http://answers.splunk.com/questions/6623/conditional-index-and-sourcetype-name-inputs-conf-by-file-n...

You just need to define a regex that will capture that value, and apply a transform to incoming events for that source or host. Something like:

# transforms.conf
[override-sourcetype]
SOURCE_KEY = source
DEST_KEY = MetaData:Sourcetype
REGEX = transactionType=\"([^\"]+)\"
FORMAT = sourcetype::$1

# props.conf
[source::/var/log/inputdir/*]
TRANSFORMS-sourcetype = override-sourcetype

For more information, take a look at:
http://www.splunk.com/base/Documentation/latest/Data/Advancedsourcetypeoverrides

View solution in original post

southeringtonp
Motivator

This is very similar to:
http://answers.splunk.com/questions/6623/conditional-index-and-sourcetype-name-inputs-conf-by-file-n...

You just need to define a regex that will capture that value, and apply a transform to incoming events for that source or host. Something like:

# transforms.conf
[override-sourcetype]
SOURCE_KEY = source
DEST_KEY = MetaData:Sourcetype
REGEX = transactionType=\"([^\"]+)\"
FORMAT = sourcetype::$1

# props.conf
[source::/var/log/inputdir/*]
TRANSFORMS-sourcetype = override-sourcetype

For more information, take a look at:
http://www.splunk.com/base/Documentation/latest/Data/Advancedsourcetypeoverrides

mattcg
Explorer

Strange I didn't see that question when I was searching. Thank you for the response, I'll give this a try.

0 Karma
Get Updates on the Splunk Community!

NEW! Log Views in Splunk Observability Dashboards Gives Context From a Single Page

Today, Splunk Observability releases log views, a new feature for users to add their logs data from Splunk Log ...

Last Chance to Submit Your Paper For BSides Splunk - Deadline is August 12th!

Hello everyone! Don't wait to submit - The deadline is August 12th! We have truly missed the community so ...

Ready, Set, SOAR: How Utility Apps Can Up Level Your Playbooks!

 WATCH NOW Powering your capabilities has never been so easy with ready-made Splunk® SOAR Utility Apps. Parse ...