Getting Data In

How to send data (JSON/CSV) from AWS s3 to Splunk at 5 minute intervals?

rsilwal7
Loves-to-Learn Lots

Hello,

I am storing data (JSON/CSV) in s3 bucket in AWS and I want to send this data into Splunk and data is updated every 5 minutes so I want to update or create a new data log in Splunk in every 5 minutes.

I am now trying by using Splunk add-on for AWS app but I don't know if it will help to send data inside the s3 bucket or not?

Can anyone tell me the right method or way to do it??

Thank you!

Labels (6)
0 Karma

anilchaithu
Builder

@rsilwal7 

You can use splunk add-on for aws to send data from aws s3 to splunk. you should use SQS based S3 approach.

If the data volume is high, you can use this route s3 -> kinesis firehose -> Splunk (using HEC)

https://docs.splunk.com/Documentation/AddOns/released/AWS/SQS-basedS3

hope this helps.

0 Karma

rsilwal7
Loves-to-Learn Lots

Thank you for your reply.

one thing I want is that: every 5 minutes I want the data in that s3 file get send to Splunk. so, can we plan the timing as well?? 

and for HEC:

I tried using HEC previously but due to the error, I couldn't solve it.

 

0 Karma
.conf21 Now Fully Virtual!
Register for FREE Today!

We've made .conf21 totally virtual and totally FREE! Our completely online experience will run from 10/19 through 10/20 with some additional events, too!