Getting Data In

Transforms not working on JSON events from Universal Forwader

tmeader
Contributor
So, I've got (unfortunately multi-line) JSON files being sent from a host to our indexers via Universal Forwarder. By using the "sourcetype=_json" in the Universal Forwarder's inputs.conf stanza, messages are making it to the indexers just fine.
 
However, I'm trying to rename the sourcetype (as well as do, what should be, some simple extractions) based on the incoming logs' "source" on the indexer side. This isn't working at all... trying to figure out why. Based on the source visible in the Splunk when looking at the events, these should be matching.
 
inputs.conf
--
[monitor:///opt/cloud-custodian-container/.../resources.json]
sourcetype = _json
crcSalt = <SOURCE>
index = cloud_custodian

props.conf (on indexers)
--
[source::.../cloud-custodian-container/.../resources.json]
TRANSFORMS-cc_change_sourcetype = cc_change_sourcetype
TRANSFORMS-cc_indexed_fields = cc_indexed_fields

transforms.conf (on indexers)
--
[cc_change_sourcetype]
REGEX = \/cloud-custodian-container\/\d{12}\/[\w\-]+\/
FORMAT = sourcetype::cloud_custodian
SOURCE_KEY = MetaData:Source
DEST_KEY = MetaData:Sourcetype

[cc_indexed_fields]
REGEX = \/cloud-custodian-container\/(\d{12})\/([\w\-]+)\/
FORMAT = aws_account_id::$1 region::$2
SOURCE_KEY = MetaData:Source
WRITE_META = true

 

Labels (3)
0 Karma

manjunathmeti
SplunkTrust
SplunkTrust

Configure props and transforms configurations on universal forwarder.

0 Karma

tmeader
Contributor

I understand that I could turn on "force_local_processing" potentially, but this should be unnecessary given this (seemingly) simple setup. Also, the EC2 instance being used to host the Universal Forwarder is intentionally trying to be kept as small sized as possible. Introducing any further overhead to the Universal Forwarder side when not required is undesirable.

0 Karma

manjunathmeti
SplunkTrust
SplunkTrust

Since you are using default sourcetype _json which is already set up with INDEXED_EXTRACTIONS=json attribute in Universal Forwarder's etc/system/default/props.conf, parsing happens in universal forwarder only. 

More info here on parsing phase:  https://docs.splunk.com/Documentation/Splunk/8.1.1/Admin/Configurationparametersandthedatapipeline#S...

https://wiki.splunk.com/images/6/63/Splunk_EventProcessing_v20_1_UF_Indexer.pdf

Instead of using transforms.conf, you can use props.conf to override automated source type matching and explicitly assign a single source type to all data coming from a specific source on universal forwarder.

[source::.../cloud-custodian-container/*/*/resources.json]
sourcetype=cloud_custodian

0 Karma
Get Updates on the Splunk Community!

Improve Your Security Posture

Watch NowImprove Your Security PostureCustomers are at the center of everything we do at Splunk and security ...

Maximize the Value from Microsoft Defender with Splunk

 Watch NowJoin Splunk and Sens Consulting for this Security Edition Tech TalkWho should attend:  Security ...

This Week's Community Digest - Splunk Community Happenings [6.27.22]

Get the latest news and updates from the Splunk Community here! News From Splunk Answers ✍️ Splunk Answers is ...