splunkd: Read operation timed out expecting ACK from x.x.x.x:9997 in 300 seconds

[tcador]

Hello,

Seeing this WARN log message roughly every 1-2 minutes on forwarders sending logs with ACK enabled to two separate indexers.

WARN TcpOutputProc - Read operation timed out expecting ACK from x.x.x.x:9997 in 300 seconds.

Also seeing these around the same time: TcpOutputProc - Possible duplication of events with channel=source::my_source_file_path|host::my_host_name|encrypted|49628, streamId=1484623843723112376, offset=32768 on host=x.x.x.x:9997

I am also seeing about 5% duplication of events.

[lguinn2]

Well, yes - you would see duplication because this message means the forwarder is saying "I am not getting the acknowledgement that is required, so I am resending the data".

So I would check the connectivity between the forwarders and this host. I would also look at the host to see what problems it is having.

[lguinn2]

Is one or more of the indexers out of disk space? Overloaded and hanging?

If it isn't a network issue, and you don't see performance problems of any kind on the server, I would open a support ticket.

[tcador]

I should have mentioned this in the original question. I have checked/verified connectivity between the forwarder and indexer and do not see any issues. Are there any configurations on either end that I could be missing?