Getting Data In

Events from multiple forwarders being dropped randomly.

caitcait
Explorer

We are experiencing random events dropping across multiple forwarders. We have a repro of the problem as we were doing some testing yesterday on one host and literally saw it drop about 20 events. Meaning these events never made it to Splunk. Events before and after these 20 events were indexed. There was no messages in splunkd on the forwarder indicating any sort of problems with 'dropped events'.
Also, these events are very similar in structure so it wouldnt be a time stamp issue.

Can anyone advise what is the best way to troubleshoot this problem? Or offer any advise as to what could be the issue?

0 Karma
1 Solution

maciep
Champion

You can probably search the internal splunk logs on the forwarder for warn or error log levels. And probably the same on the indexer. For example, looking for connection errors from the forwarder to the indexer.

But do you have acknowledgement turned on, so if the forwarder sends an event to the indexer it waits for an ack before clearing it from the queue?

Anything similar about the missing events that might differentiate them? Different log files? Different sourcetypes? etc?

View solution in original post

0 Karma

maciep
Champion

You can probably search the internal splunk logs on the forwarder for warn or error log levels. And probably the same on the indexer. For example, looking for connection errors from the forwarder to the indexer.

But do you have acknowledgement turned on, so if the forwarder sends an event to the indexer it waits for an ack before clearing it from the queue?

Anything similar about the missing events that might differentiate them? Different log files? Different sourcetypes? etc?

0 Karma
Get Updates on the Splunk Community!

Take Your Breath Away with Splunk Risk-Based Alerting (RBA)

WATCH NOW!The Splunk Guide to Risk-Based Alerting is here to empower your SOC like never before. Join Haylee ...

SignalFlow: What? Why? How?

What is SignalFlow? Splunk Observability Cloud’s analytics engine, SignalFlow, opens up a world of in-depth ...

Federated Search for Amazon S3 | Key Use Cases to Streamline Compliance Workflows

Modern business operations are supported by data compliance. As regulations evolve, organizations must ...