We have decided to use S3 as centralized collection of logs from various sources. I have the following questions to have have the best way to get data into Splunk from S3. Daily ingestion of logs will be around 5 Gb. There will be multiple inputs needs to be created to read each s3 buckets.
1. I see there are 2 Addon's available. Splunk Add-on for AWS and Splunk Add-on for Amazon Kinesis Firehose. Will both Add-on supports reading data from s3? If so which is the best method?
2. Splunk Add on Aws has 3 input types : Generic S3 , Incremental S3 , SQS based S3. Which will be the best method to ingest data from multiple directories and around 5Gb per day.