Getting Data In

SQS based S3 input is skipping some objects from s3 but deleting the message from sqs?

srinikrishna
New Member

Hi 

We have a splunk add-on for aws to pull the logs from s3 bucket. we are using the sqs based s3 inputs created to read the logs for s3 bucket, however we are noticing that through this option splunk service seems like ommiting some files from reading even though it has consumed the sqs message and deleted the message from the quer. I am attaching one of the example from our issue where in at particular time frame which is on july 6th 19 to 20 hrs we have 59 objects in the s3 bucket but splunk had read only 58 files. This is being one of the example to show but we are having this issue very often every hour one or 2 files missing. We have around 8000 to 10000 events in each file which is missing indexed in splunk due to this issue. I have checked all the internal logs which does not show any failure messages while reading this particular s3 object to confirm it was dropped or failed while parsing and processing. Its just not there.  these issue is there every day every hour missing one or the other files missed by splunk inputs. From SQS perspective SQS based S3 input is skipping some objects from s3 but deleting the message from sqs

s3bucketobject.jpgsplunkmissingobject.jpg

Thanks and Regards Srini

Labels (1)
Tags (2)
0 Karma

soumdey0192
Explorer

Hello @srinikrishna ,

Did you get a solution for this issue that you reported earlier?

As I have a similar issue with one of my use case.

For ref - https://community.splunk.com/t5/Getting-Data-In/JSON-files-are-not-being-onboarded-intermittently-vi...

Thanks

 

0 Karma
Get Updates on the Splunk Community!

Take Your Breath Away with Splunk Risk-Based Alerting (RBA)

WATCH NOW!The Splunk Guide to Risk-Based Alerting is here to empower your SOC like never before. Join Haylee ...

Industry Solutions for Supply Chain and OT, Amazon Use Cases, Plus More New Articles ...

Splunk Lantern is a Splunk customer success center that provides advice from Splunk experts on valuable data ...

Enterprise Security Content Update (ESCU) | New Releases

In November, the Splunk Threat Research Team had one release of new security content via the Enterprise ...