All Apps and Add-ons

AWS SQS Based S3 input - ignore historical data - start ingest from today

danan5
Path Finder

Hi all,

I have an issue with a new SQS Based S3 input. There is a large amount of historical data in S3 that I don't want to ingest.

Is there a way to manipulate the pointer used by Splunk when reading S3 so it starts ingesting data from a nominated time, for example today onwards and thereby ignore historical data?

Many thanks,
David

Tags (2)
0 Karma

srinikrishna
New Member

Hi, 

with the generic s3 input, you do have option of giving the start date and time and the end date and time. You can use that to filter the unwanted objects to be read from s3. However if you are asking for sqs based s3, you can setup the events in s3 bucket to create the sqs message on new create object only which means only from next new object placed in s3 bucket you will have the message. with sqs based s3 this is the only option you have to set on what event you need the message to be sent. if you want to read from particular time, you need to use the generic s3 inputs. 

srinikrishna_0-1594193719534.png

 

Tags (1)
0 Karma
Get Updates on the Splunk Community!

Happy CX Day, Splunk Community!

Happy CX Day, Splunk Community! CX stands for Customer Experience, and today, October 3rd, is CX Day — a ...

Splunk Observability Cloud | Customer Survey!

If you use Splunk Observability Cloud, we invite you to share your valuable insights with us through a brief ...

.conf23 | Get Your Cybersecurity Defense Analyst Certification in Vegas

We’re excited to announce a new Splunk certification exam being released at .conf23! If you’re going to Las ...