Transforms working intermittently on index cluster for AWS Kinesis to HEC

[johnansett]

Hello Splunkers,

We are receiving config notifications, CloudTrail and others from AWS through Kinesis - the general pattern is:
Config -> Event Rules -> Event Hub -> Kinesis -> HEC (indexer cluster)
This works, what seems flawlessly.

We separate the data into indexes based on the account number due to security policies so I've set up props/transforms to do this.

This seems to work most of the time and 77% of the traffic ends up in the correct index but there is 33% (on avg) that ends up in the default index of the HEC (aws_config)

Is it possible that the transforms aren't triggering all the time?
The events are identical format; sourcetype and source are identical.

Here is the transforms:

[aws-account1]
REGEX = 010016492034
DEST_KEY = _MetaData:Index
FORMAT = aws-account1

Props:
[aws:config:notification]
TRANSFORMS-aws_config_notification=aws-account1

Am I missing something here? Is there anything I should look for in internal?

I remember years ago in training an example where props hierarchy would mess with data when there was multiple props/transforms, and the intermittent nature *might* make sense but I have no idea where to troubleshoot this.
The HEC is on a cluster of indexers so the config is all via the CM, thus no differences.

Any suggestions would be greatly appreciated!

Cheers!

[johnansett · Accepted solution]

Regex issue.  Multiple account numbers in events.