Splunk Cloud Dropping Events

[cyberbilliam]

This is more of an advisory than a question.  I hope it helps.

If you are a Splunk Cloud customer I strongly suggest you run this search to ensure that Splunk Cloud is not dropping events.  This info is not being presented in the Splunk Cloud monitoring console and is an indicator that indexed events are being dropped.

index=_internal host=idx* sourcetype=splunkd log_level IN(ERROR,WARN) component=SQSSmartbusInputWorker "Error parsing events from message content"
| eval bytesRemaining=trim(bytesRemaining,":")
| stats sum(bytesRemaining) as bytesNotIndexed

What these errors are telling us is that some SQSSmartbusInputWorker process is parsing events and that there is some type of invalid field, or value in the data, in our case _subsecond.  When this process hits the invalid value, it appears to drop everything else in the stream (i.e. bytesRemaining).  So this is also to say that bytesRemaining contains events that were sent to Splunk Cloud, but not indexed.  

When this error occurs,  Splunk cloud writes the failed info to an SQS DLQ in S3 which can be observed using:

index=_internal host=idx* sourcetype=splunkd log_level IN(ERROR,WARN) component=SQSSmartbusInputWorker "Successfully sent a SQS DLQ message to S3 with location"

Curious if anyone else out there is experiencing the same issue.  SQSSmartbusInputWorker  doesn't appear in any of the indexing documents, but does appear to be very important to the ingest process.

[new_splunker]

@cyberbilliam 
Is this fixed? Need confirmation before migrating to Splunk Cloud.

[mike_fisher]

In general, it should be.  That said, make sure you don't try to send events via HEC to an undefined index in Splunk Cloud.  You will likely lose any events after the one destined for the undefined index in a given HEC batch.