Why this error after upgrade to 9.0 "ERROR TcpOutputQ [<thread id> TcpOutEloop] - Unexpected event id=<eventid>"?

[hrawat_splunk]

After upgrade to 9.0 seeing following

ERROR TcpOutputQ [<thread id> TcpOutEloop] - Unexpected event id=<eventid>

[vinayakwagh]

It helps Thanks

[hrawat_splunk · Accepted solution]

If useACK set to true and batch mode is on(default on) with Splunk 9.0, there is a possibility of hitting following error log messages.

"Unexpected event id"
"Invalid ACK received from indexer"
"Got unexpected ACK with eventid"

This may also lead to blocked queues on forwarding tier.

autoLBVolume and autoBatch while processing an event, apply limit using raw size of the event. However if there are  raw less events ( e.g. metrics events) autoLBVolume and autoBatch will end up sending lot more events then configured limits to receiver.
With autoLBVolume, it results in more than expected/configured events distributed to receivers.

With autoBatch, it results in batch of lot more events than expected. That means while a batch of thousands of events being sent to receiver, at the same time some events are already getting acknowledged.
Forwarder creates a list of  events to be acknowledged after successfully sending batch of events. However if the batch is in-flight over TCP layer and forwarder receives an ACKed event of the batch, it's not in the list of expected events to be acknowledged.  That leads to above ERROR.

Workaround: Either set useACK=false or autoBatch=false

Issue is fixed by 9.0.3 patch.

Note: 
After 9.0.3 upgrade, you will still see benign “Unexpected event id” log message. However there should not be following log messages.
"Invalid ACK received from indexer"
"Got unexpected ACK with eventid"

[woodcock]

It is back in v9.0.1

[hrawat_splunk]

See my updated answer. 9.0.1 still logs the ERROR, but it does not block forwarder.

[Sithima]

If the issue is fixed in 9.0.1, why am I getting the same error message in Splunk 9.0.1?

ERROR TcpOutputQ [<id> TcpOutEloop] - Unexpected eventid=<id>

[hrawat_splunk]

9.0.1 has not suppressed the ERROR log. It fixes the underlying tcpout queue blockage  issue. While you see the ERROR log but no tcpout queue blockage (as seen with 9.0.0) is an indication that the tcpout queue blockage  issue is resolved.

Will suppress 9.0.1 benign ERROR log in future releases. 

[woodcock]

Actually the problem is still there.  I was getting continuous crashes on my HWF.

[hrawat_splunk]

That crash is still an issue and will be fixed.  It happens if forceTimebasedAutoLB=true
Workaround for 9.0.1 TcpOutputQ crash 
Set one of the following

forceTimebasedAutoLB=false

or

autoBatch=false

or

connectionsPerTarget=1

This crash is applicable if UF/HF resolves < 10 target IP addresses and forceTimebasedAutoLB=true.