Getting Data In

How to send data (JSON/CSV) from AWS s3 to Splunk at 5 minute intervals?

rsilwal7
Loves-to-Learn Lots

Hello,

I am storing data (JSON/CSV) in s3 bucket in AWS and I want to send this data into Splunk and data is updated every 5 minutes so I want to update or create a new data log in Splunk in every 5 minutes.

I am now trying by using Splunk add-on for AWS app but I don't know if it will help to send data inside the s3 bucket or not?

Can anyone tell me the right method or way to do it??

Thank you!

Labels (6)
0 Karma

anilchaithu
Builder

@rsilwal7 

You can use splunk add-on for aws to send data from aws s3 to splunk. you should use SQS based S3 approach.

If the data volume is high, you can use this route s3 -> kinesis firehose -> Splunk (using HEC)

https://docs.splunk.com/Documentation/AddOns/released/AWS/SQS-basedS3

hope this helps.

0 Karma

rsilwal7
Loves-to-Learn Lots

Thank you for your reply.

one thing I want is that: every 5 minutes I want the data in that s3 file get send to Splunk. so, can we plan the timing as well?? 

and for HEC:

I tried using HEC previously but due to the error, I couldn't solve it.

 

0 Karma
Get Updates on the Splunk Community!

Happy CX Day to our Community Superheroes!

Happy 10th Birthday CX Day!What is CX Day? It’s a global celebration recognizing innovation and success in the ...

Check out This Month’s Brand new Splunk Lantern Articles

Splunk Lantern is a customer success center providing advice from Splunk experts on valuable data insights, ...

Routing Data to Different Splunk Indexes in the OpenTelemetry Collector

This blog post is part of an ongoing series on OpenTelemetry. The OpenTelemetry project is the second largest ...