How to send data (JSON/CSV) from AWS s3 to Splunk at 5 minute intervals?

[rsilwal7]

Hello,

I am storing data (JSON/CSV) in s3 bucket in AWS and I want to send this data into Splunk and data is updated every 5 minutes so I want to update or create a new data log in Splunk in every 5 minutes.

I am now trying by using Splunk add-on for AWS app but I don't know if it will help to send data inside the s3 bucket or not?

Can anyone tell me the right method or way to do it??

Thank you!

[anilchaithu]

@rsilwal7 

You can use splunk add-on for aws to send data from aws s3 to splunk. you should use SQS based S3 approach.

If the data volume is high, you can use this route s3 -> kinesis firehose -> Splunk (using HEC)

https://docs.splunk.com/Documentation/AddOns/released/AWS/SQS-basedS3

hope this helps.

[rsilwal7]

Thank you for your reply.

one thing I want is that: every 5 minutes I want the data in that s3 file get send to Splunk. so, can we plan the timing as well?? 

and for HEC:

I tried using HEC previously but due to the error, I couldn't solve it.