All Apps and Add-ons
Highlighted

AWS SQS Based S3 input - ignore historical data - start ingest from today

Explorer

Hi all,

I have an issue with a new SQS Based S3 input. There is a large amount of historical data in S3 that I don't want to ingest.

Is there a way to manipulate the pointer used by Splunk when reading S3 so it starts ingesting data from a nominated time, for example today onwards and thereby ignore historical data?

Many thanks,
David

Tags (2)
0 Karma
Highlighted

Re: AWS SQS Based S3 input - ignore historical data - start ingest from today

New Member

Hi, 

with the generic s3 input, you do have option of giving the start date and time and the end date and time. You can use that to filter the unwanted objects to be read from s3. However if you are asking for sqs based s3, you can setup the events in s3 bucket to create the sqs message on new create object only which means only from next new object placed in s3 bucket you will have the message. with sqs based s3 this is the only option you have to set on what event you need the message to be sent. if you want to read from particular time, you need to use the generic s3 inputs. 

srinikrishna_0-1594193719534.png

 

Tags (1)
0 Karma
Speak Up for Splunk Careers!

We want to better understand the impact Splunk experience and expertise has has on individuals' careers, and help highlight the growing demand for Splunk skills.