Getting Data In

SQS based S3 input is skipping some objects from s3 but deleting the message from sqs?

srinikrishna
New Member

Hi 

We have a splunk add-on for aws to pull the logs from s3 bucket. we are using the sqs based s3 inputs created to read the logs for s3 bucket, however we are noticing that through this option splunk service seems like ommiting some files from reading even though it has consumed the sqs message and deleted the message from the quer. I am attaching one of the example from our issue where in at particular time frame which is on july 6th 19 to 20 hrs we have 59 objects in the s3 bucket but splunk had read only 58 files. This is being one of the example to show but we are having this issue very often every hour one or 2 files missing. We have around 8000 to 10000 events in each file which is missing indexed in splunk due to this issue. I have checked all the internal logs which does not show any failure messages while reading this particular s3 object to confirm it was dropped or failed while parsing and processing. Its just not there.  these issue is there every day every hour missing one or the other files missed by splunk inputs. From SQS perspective SQS based S3 input is skipping some objects from s3 but deleting the message from sqs

s3bucketobject.jpgsplunkmissingobject.jpg

Thanks and Regards Srini

Labels (1)
Tags (2)
0 Karma

soumdey0192
Explorer

Hello @srinikrishna ,

Did you get a solution for this issue that you reported earlier?

As I have a similar issue with one of my use case.

For ref - https://community.splunk.com/t5/Getting-Data-In/JSON-files-are-not-being-onboarded-intermittently-vi...

Thanks

 

0 Karma
Get Updates on the Splunk Community!

How to Monitor Google Kubernetes Engine (GKE)

We’ve looked at how to integrate Kubernetes environments with Splunk Observability Cloud, but what about ...

Index This | How can you make 45 using only 4?

October 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with this ...

Splunk Education Goes to Washington | Splunk GovSummit 2024

If you’re in the Washington, D.C. area, this is your opportunity to take your career and Splunk skills to the ...