HEC Sourcetype Transforms

jondukehds — Tue, 07 Jul 2020 18:06:08 GMT

I am trying to split HEC data into multiple sourcetype based on regex. The Docker platform we are using only provides three inputs for sending data to Splunk this way per group of servers.

Host

Port

Splunk Token

So I cannot define by token what the sourcetype is. I also have zero access to that interface to play around with it. In short, the data all has to come in on the same token.

Now to the question.

Referencing various answers on this site, along with the .spec splunk documentation for props.conf and transforms.conf.

This one.

https://community.splunk.com/t5/Getting-Data-In/How-to-split-a-log-into-multiple-sourcetypes-on-a-heavy/td-p/173635

and

https://community.splunk.com/t5/Getting-Data-In/How-to-ingest-logs-from-a-single-file-and-split-them-into-2/td-p/256349

I am trying to, by REGEX pattern, split this one HEC sourcetype "mule_system" into multiple sourcetypes on my heavy forwarder.

I have had some success, but with a weird side-effect. The transforms works, and in a real-time (30sec) search of the data appears in search as two sourcetypes.

On a longer running search of this same event, (15m) it reverts back to the sourcetype defined in the inputs.conf for this hec token. "mule_system"

Here are the config files I am using on my heavy forwarder. Keep in mind that more transforms statements will be inserted as we categorize the events into sourcetypes.

Inputs.conf

[http://mule_hec] disabled = 0 index = sandbox sourcetype = mule_system token = b7483125-9d88-49b9-bd90-xxxxxxxxxx source = mule_hec

props.conf

[source::mule_hec] #all matches everything for testing #TRANSFORMS-changeSourcetype = mule-app-all TRANSFORMS-changeSourcetype1 = mule-app-setsourcetype TRANSFORMS-changeSourcetype2 = mule-http-request-setsourcetype

transforms.conf

#testing to see if transforms is working at all. [mule-app-all] DEST_KEY = MetaData::Sourcetype REGEX = .*? FORMAT = sourcetype::mule-rtf WRITE_META = true [mule-app-setsourcetype] DEST_KEY = MetaData::Sourcetype REGEX = (LoggerMessageProcessor) FORMAT = sourcetype::mule-app WRITE_META = true [mule-http-request-setsourcetype] DEST_KEY = MetaData::Sourcetype #REGEX = msg=\"http request\" REGEX = user-agent FORMAT = sourcetype::mule-http-request WRITE_META = true

Re: HEC Sourcetype Transforms

jondukehds — Fri, 10 Jul 2020 19:32:01 GMT

The issue here was in the transforms. One too many colons.

DEST_KEY = MetaData::Sourcetype

Should have been

DEST_KEY = MetaData:Sourcetype

topic HEC Sourcetype Transforms in Getting Data In

HEC Sourcetype Transforms

Re: HEC Sourcetype Transforms