topic Re: AWS to Splunk in Getting Data In

How to send data (JSON/CSV) from AWS s3 to Splunk at 5 minute intervals?

rsilwal7 — Mon, 07 Sep 2020 17:21:17 GMT

Hello,

I am storing data (JSON/CSV) in s3 bucket in AWS and I want to send this data into Splunk and data is updated every 5 minutes so I want to update or create a new data log in Splunk in every 5 minutes.

I am now trying by using Splunk add-on for AWS app but I don't know if it will help to send data inside the s3 bucket or not?

Can anyone tell me the right method or way to do it??

Thank you!

Re: AWS to Splunk

anilchaithu — Sat, 05 Sep 2020 23:12:50 GMT

@rsilwal7

You can use splunk add-on for aws to send data from aws s3 to splunk. you should use SQS based S3 approach.

If the data volume is high, you can use this route s3 -> kinesis firehose -> Splunk (using HEC)

https://docs.splunk.com/Documentation/AddOns/released/AWS/SQS-basedS3

hope this helps.

Re: AWS to Splunk

rsilwal7 — Sun, 06 Sep 2020 08:11:57 GMT

Thank you for your reply.

one thing I want is that: every 5 minutes I want the data in that s3 file get send to Splunk. so, can we plan the timing as well??

and for HEC:

I tried using HEC previously but due to the error, I couldn't solve it.