AWS SQS Based S3 input - ignore historical data - start ingest from today

[danan5]

Hi all,

I have an issue with a new SQS Based S3 input. There is a large amount of historical data in S3 that I don't want to ingest.

Is there a way to manipulate the pointer used by Splunk when reading S3 so it starts ingesting data from a nominated time, for example today onwards and thereby ignore historical data?

Many thanks,
David

[srinikrishna]

Hi, 

with the generic s3 input, you do have option of giving the start date and time and the end date and time. You can use that to filter the unwanted objects to be read from s3. However if you are asking for sqs based s3, you can setup the events in s3 bucket to create the sqs message on new create object only which means only from next new object placed in s3 bucket you will have the message. with sqs based s3 this is the only option you have to set on what event you need the message to be sent. if you want to read from particular time, you need to use the generic s3 inputs.