Too many streaming errors to target on cluster

[responsys_cm]

I've got a cluster with three identical indexers. One indexer consistently generates the "Too many streaming errors to target". I've checked the network adapter on the server and there are zero errors over 4 TB of traffic. I've tried increasing the error count from 3 to 6, but it still happens.

How do I figure out the root cause of this issue?

Thanks.

Craig

[svasan_splunk]

Were there any replication/tcp related errors in the target node's splunkd.log?

[svasan_splunk]

That could be the cause of your problem. Though the BucketReplicator errors on the originating node not on the target of the replication. When it says "Too many streaming errors" that's anything that causes the replication to fail. Do you see why the writes fail? Did they just timeout?

Also when you say you checked the network adapter, what did you check?

[responsys_cm]

WARN BucketReplicator - Failed to replicate warm bucket _internal~323~3B990028-D1F8-40B1-B39C-DD13D01D8FAF to E490E3C8-F237-490F-A245-082AE868B4F8. Replication to 10.230.226.21:8090:_internal~323~3B990028-D1F8-40B1-B39C-DD13D01D8FAF failed. Write failed.

ERROR BucketReplicator - Replication to 10.230.226.21:8090:_internal~323~3B990028-D1F8-40B1-B39C-DD13D01D8FAF failed. Write failed.

[responsys_cm]

These are the three error messages I see a ton of:

CMSlave - bid=_internal~323~3B990028-D1F8-40B1-B39C-DD13D01D8FAF src=3B990028-D1F8-40B1-B39C-DD13D01D8FAF tgt=E490E3C8-F237-490F-A245-082AE868B4F8 failing=E490E3C8-F237-490F-A245-082AE868B4F8 queued replication error job