Getting Data In

Transforms working intermittently on index cluster for AWS Kinesis to HEC

johnansett
Communicator

Hello Splunkers,

We are receiving config notifications, CloudTrail and others from AWS through Kinesis - the general pattern is:
Config -> Event Rules -> Event Hub -> Kinesis -> HEC (indexer cluster)
This works, what seems flawlessly.

We separate the data into indexes based on the account number due to security policies so I've set up props/transforms to do this.

This seems to work most of the time and 77% of the traffic ends up in the correct index but there is 33% (on avg) that ends up in the default index of the HEC (aws_config)

Is it possible that the transforms aren't triggering all the time?
The events are identical format; sourcetype and source are identical.

Here is the transforms:

[aws-account1]
REGEX = 010016492034
DEST_KEY = _MetaData:Index
FORMAT = aws-account1

Props:
[aws:config:notification]
TRANSFORMS-aws_config_notification=aws-account1

Am I missing something here? Is there anything I should look for in internal?

I remember years ago in training an example where props hierarchy would mess with data when there was multiple props/transforms, and the intermittent nature *might* make sense but I have no idea where to troubleshoot this.
The HEC is on a cluster of indexers so the config is all via the CM, thus no differences.

Any suggestions would be greatly appreciated!

Cheers!

1 Solution

johnansett
Communicator

Regex issue.  Multiple account numbers in events.

View solution in original post

0 Karma

johnansett
Communicator

Regex issue.  Multiple account numbers in events.

0 Karma
Get Updates on the Splunk Community!

Stay Connected: Your Guide to July Tech Talks, Office Hours, and Webinars!

What are Community Office Hours?Community Office Hours is an interactive 60-minute Zoom series where ...

Updated Data Type Articles, Anniversary Celebrations, and More on Splunk Lantern

Splunk Lantern is a Splunk customer success center that provides advice from Splunk experts on valuable data ...

A Prelude to .conf25: Your Guide to Splunk University

Heading to Boston this September for .conf25? Get a jumpstart by arriving a few days early for Splunk ...