AWS Landing Zone - Centralize logging - how are others ingesting?

[t9445]

Hello, hoping others may have run into this and figured out best-practice (or best-way...)

We are implementing an AWS Landing Zone, and the centralized logging for all-accounts/sourcetypes/etc is all to one S3 location, as opposed to ingesting from various sources (S3 Buckets, CloudWatch, ...) - e.g. 'Splunk App for AWS'

What we are wondering is how have others using an AWS Landing Zone, are ingesting with Splunk?
e.g At first glimpse we will have to do a lot of props/transforms to get the data broken out as it should be in Splunk if we simply ingest the S3 (as S3 or Kinesis FireHose, etc) -- props/transforms to breakout the details: e.g. Account=012345, sourcetype=cloudtrail, zone=us-east-1, etc

Is there a best/better way that folks are using for splunk-ingesting the centralized logging in the "AWS Landing Zone" ?

Any pointers appreciated (Splunk App for AWS would be kludgy for this from what we can tell, since multiple sourcetypes/accounts etc in the same-data-feed/broken-out into multiple data-feeds)

thanks

-tom

[johnansett]

Hey Tom, wondering where you ended up with this.  We are going the same route and looking at the following:

1) Log all applicable events to central S3 bucket

2) Lambda function to split into multiple queues

3) Props/transforms to send to specific indexes (required for each account)

This was posted after your original post, but might be of some use depending on where you are with the deployment:

https://www.splunk.com/en_us/blog/cloud/making-the-collection-of-centralised-s3-logs-into-splunk-eas...

[mtranchita]

My organization uses this model, logs from multiple AWS Accounts are centralized into a single AWS Account purpose built for aggregation. I've found using the Splunk App for AWS to be relatively simple to implement. The key was understanding, and making use of, prefixes and multiple buckets. YMMV but I would suggest testing things.
Hope that helps...

[dbot2001]

Are you using AWS Cloudwatch agent to forward application logs to the centralized location?