All Apps and Add-ons

AWS SQS Based S3 input - ignore historical data - start ingest from today

Path Finder

Hi all,

I have an issue with a new SQS Based S3 input. There is a large amount of historical data in S3 that I don't want to ingest.

Is there a way to manipulate the pointer used by Splunk when reading S3 so it starts ingesting data from a nominated time, for example today onwards and thereby ignore historical data?

Many thanks,

Tags (2)
0 Karma

New Member


with the generic s3 input, you do have option of giving the start date and time and the end date and time. You can use that to filter the unwanted objects to be read from s3. However if you are asking for sqs based s3, you can setup the events in s3 bucket to create the sqs message on new create object only which means only from next new object placed in s3 bucket you will have the message. with sqs based s3 this is the only option you have to set on what event you need the message to be sent. if you want to read from particular time, you need to use the generic s3 inputs. 



Tags (1)
0 Karma
Get Updates on the Splunk Community!

Happy CX Day, Splunk Community!

Happy CX Day, Splunk Community! CX stands for Customer Experience, and today, October 3rd, is CX Day — a ...

Splunk Observability Cloud | Customer Survey!

If you use Splunk Observability Cloud, we invite you to share your valuable insights with us through a brief ...

.conf23 | Get Your Cybersecurity Defense Analyst Certification in Vegas

We’re excited to announce a new Splunk certification exam being released at .conf23! If you’re going to Las ...