What does "PollableDescriptor:writeWithTimeout failed" error mean?

[the_wolverine]

Seeing the following error in LWF splunkd.log every 5 minutes:

10-28-2010 08:37:37.048 WARN  NetUtils - PollableDescriptor::writeWithTimeout failed. Bad select_for_loop rv = -2
10-28-2010 08:37:37.048 WARN  TcpOutputProc - TcpSendThread: Connection to server the_indexer_ip:9443, fd:33 lost - retrying: Resource temporarily unavailable

What does it mean? Data is coming through, but there's a serious lag (1 hour) at times. Currently it is caught up so seems to be an intermittent problem.

[jkerai · Accepted solution]

This warning is logged on forwarder when it is unable to write data to network for more than 5 mins. This may be why you are seeing the lag. Please check the 'current_size' of queues on the indexer to see if they are always near their max capacity of 1000 events around the time this warning is seen. If the queues on indexer look healthy, then check to see if there network issue during the time this warning is reported. You can probably look for the same warning on other forwarders to see if other forwarders also experienced the same problem. May be indicative of network or indexer saturation.