AWS Logs via Lamda to HF drop preceding key from fieldname

cybersecnutant — Wed, 03 Jan 2024 21:25:46 GMT

Hello, I've got a Lamda function exporting AWS logs via HEC to my HF's to my indexers.

Unfortunately, the AWS logs are coming in with event.* as all of the field names, whereas the Splunk_TA_aws addon is expecting *.

I can easily do a rename event.* as a *, however that's too late for the out of the box props.conf's to take effect. This causes things like the the

"FIELDALIAS-eventName-for-aws-cloudtrail-command = eventName AS commandrename eventName as command"

in props.conf to fail unless I go in and modify it to be event.eventName. I'd like to fix this before it gets to SPL. Is there a way to do this easily? Thanks!

Re: AWS Logs via Lamda to HF drop preceding key from fieldname

tscroggins — Sat, 06 Jan 2024 17:07:24 GMT

Hi @cybersecnutant,

If we assume your events are JSON-formatted:

{"event.field1": "value1", "event.field2": "value2", "event.field3": "value3"}

you can remove the event. field prefix using SEDCMD in props.conf on your heavy forwarder:

# local/props.conf [your_sourcetype] SEDCMD-strip-event = s/"event\.([^"]+)"/"\1"/g

or a combination of RULESET in props.conf and a transform in transforms.conf (together, an ingest action) on either your heavy forwarder or your indexer(s):

# local/props.conf [your_sourcetype] RULESET-strip-event = strip-event # local/transforms.conf [strip-event] INGEST_EVAL = _raw:=replace(_raw, "\"event\.([^\"]+)\"", "\"\\1\"")

You can also reference the same INGEST_EVAL transform in a TRANSFORMS setting in props.conf. The difference is where in the pipeline the various methods execute. SEDCMD and TRANSFORMS execute between typingQueue and rulesetQueue (along the path for "raw" data), and RULESET executes between rulesetQueue and indexQueue (the injection point for parsed or "cooked" data).

Your final raw event would be:

{"field1": "value1", "field2": "value2", "field3": "value3"}

topic AWS Logs via Lamda to HF drop preceding key from fieldname in Getting Data In

AWS Logs via Lamda to HF drop preceding key from fieldname

Re: AWS Logs via Lamda to HF drop preceding key from fieldname