topic Re: Why this error after upgrade to 9.0 "ERROR TcpOutputQ [<thread id> TcpOutEloop] - Unexpected event id=&l in Getting Data In

Why this error after upgrade to 9.0 "ERROR TcpOutputQ [ TcpOutEloop] - Unexpected event id="?

hrawat — Sat, 09 Jul 2022 16:46:30 GMT

After upgrade to 9.0 seeing following

ERROR TcpOutputQ [<thread id> TcpOutEloop] - Unexpected event id=<eventid>

Re: Why this error after upgrade to 9.0 "ERROR TcpOutputQ [ TcpOutEloop] - Unexpected event id=&l

hrawat — Wed, 18 Jan 2023 04:07:44 GMT

If useACK set to true and batch mode is on(default on) with Splunk 9.0, there is a possibility of hitting following error log messages.

"Unexpected event id"
"Invalid ACK received from indexer"
"Got unexpected ACK with eventid"

This may also lead to blocked queues on forwarding tier.

autoLBVolume and autoBatch while processing an event, apply limit using raw size of the event. However if there are raw less events ( e.g. metrics events) autoLBVolume and autoBatch will end up sending lot more events then configured limits to receiver.
With autoLBVolume, it results in more than expected/configured events distributed to receivers.

With autoBatch, it results in batch of lot more events than expected. That means while a batch of thousands of events being sent to receiver, at the same time some events are already getting acknowledged.
Forwarder creates a list of events to be acknowledged after successfully sending batch of events. However if the batch is in-flight over TCP layer and forwarder receives an ACKed event of the batch, it's not in the list of expected events to be acknowledged. That leads to above ERROR.

Workaround: Either set useACK=false or autoBatch=false

Issue is fixed by 9.0.3 patch.

Note:
After 9.0.3 upgrade, you will still see benign “Unexpected event id” log message. However there should not be following log messages.
"Invalid ACK received from indexer"
"Got unexpected ACK with eventid"

Re: Why this error after upgrade to 9.0 "ERROR TcpOutputQ [ TcpOutEloop] - Unexpected event id=&l

vinayakwagh — Fri, 15 Jul 2022 09:04:21 GMT

It helps Thanks

Re: Why this error after upgrade to 9.0 "ERROR TcpOutputQ [ TcpOutEloop] - Unexpected event id=&l

Sithima — Thu, 06 Oct 2022 05:05:49 GMT

If the issue is fixed in 9.0.1, why am I getting the same error message in Splunk 9.0.1?

ERROR TcpOutputQ [<id> TcpOutEloop] - Unexpected eventid=<id>

Re: Why this error after upgrade to 9.0 "ERROR TcpOutputQ [ TcpOutEloop] - Unexpected event id=&a

hrawat — Thu, 06 Oct 2022 12:56:46 GMT

9.0.1 has not suppressed the ERROR log. It fixes the underlying tcpout queue blockage issue. While you see the ERROR log but no tcpout queue blockage (as seen with 9.0.0) is an indication that the tcpout queue blockage issue is resolved.

Will suppress 9.0.1 benign ERROR log in future releases.

Re: Why this error after upgrade to 9.0 "ERROR TcpOutputQ [ TcpOutEloop] - Unexpected event id=&l

woodcock — Tue, 01 Nov 2022 15:05:53 GMT

It is back in v9.0.1

Re: Why this error after upgrade to 9.0 "ERROR TcpOutputQ [ TcpOutEloop] - Unexpected event id=&a

hrawat — Tue, 01 Nov 2022 15:17:37 GMT

See my updated answer. 9.0.1 still logs the ERROR, but it does not block forwarder.

Re: Why this error after upgrade to 9.0 "ERROR TcpOutputQ [ TcpOutEloop] - Unexpected event id=&a

woodcock — Tue, 01 Nov 2022 18:36:39 GMT

Actually the problem is still there. I was getting continuous crashes on my HWF.

Re: Why this error after upgrade to 9.0 "ERROR TcpOutputQ [ TcpOutEloop] - Unexpected event id=&a

hrawat — Tue, 01 Nov 2022 19:22:40 GMT

That crash is still an issue and will be fixed. It happens if forceTimebasedAutoLB=true
Workaround for 9.0.1 TcpOutputQ crash
Set one of the following

forceTimebasedAutoLB=false

autoBatch=false

connectionsPerTarget=1

This crash is applicable if UF/HF resolves < 10 target IP addresses and forceTimebasedAutoLB=true.

Re: Why this error after upgrade to 9.0 "ERROR TcpOutputQ [ TcpOutEloop] - Unexpected event id=&a

pmerlin1 — Fri, 22 Dec 2023 15:04:26 GMT

The issue is not fixed after upgrading 9.1.2. This issue occured on search head cluster.

My settings in outputs.conf :

[indexer_discovery:target_master] pass4SymmKey = ********** [tcpout] defaultGroup = default_indexers forceTimebasedAutoLB = true maxQueueSize = 7MB useACK = true [tcpout:default_indexers] server = **********01:9997,**********02.lan:9997

Re: Why this error after upgrade to 9.0 "ERROR TcpOutputQ [ TcpOutEloop] - Unexpected event id=&a

Vwagh — Fri, 22 Dec 2023 15:04:42 GMT

It needs to be

UseAck = false

Then this errors should resolve.

Re: Why this error after upgrade to 9.0 "ERROR TcpOutputQ [ TcpOutEloop] - Unexpected event id=&a

hrawat — Fri, 22 Dec 2023 21:10:31 GMT

Following three logs

"Unexpected event id" ( 9.1.2 still logs)
"Invalid ACK received from indexer" ( 9.1.2 should not log)
"Got unexpected ACK with eventid" (9.1.2 should not log)

What exactly the issue you are hitting?

Re: Why this error after upgrade to 9.0 "ERROR TcpOutputQ [ TcpOutEloop] - Unexpected event id=&l

anandhalagaras1 — Thu, 08 Feb 2024 12:07:37 GMT

@lawrence_magpoc ,

I am running with Splunk Universal Forwarder 9.0.2 in one of my Linux client machine and recently for the past couple of days i am getting this events in the internal logs and it seems like its getting crashed and once again the service is getting started automatically.

[build 17e00c557dc1] 2024-02-08 05:26:15 Received fatal signal 6 (Aborted) on PID 1908113. Cause: Signal sent by PID 1908113 running under UID 9991. Crashing thread: TcpOutEloop Registers: RIP: [0x00007F65EB39AACF] gsignal + 271 (libc.so.6 + 0x4EACF)

ERROR TcpOutputQ [1908232 TcpOutEloop] - Unexpected event id=30

ERROR TcpOutputQ [1908232 TcpOutEloop] - Unexpected event id=29

So how to fix this issue and also in which config file we need to add in the client machine where UF is running.

autoBatch=false

Re: Why this error after upgrade to 9.0 "ERROR TcpOutputQ [ TcpOutEloop] - Unexpected event id=&a

hrawat — Thu, 08 Feb 2024 13:09:43 GMT

In outputs.conf set

useACK=false autoBatch=false

Re: Why this error after upgrade to 9.0 "ERROR TcpOutputQ [ TcpOutEloop] - Unexpected event id=&a

pmerlin1 — Thu, 08 Feb 2024 15:37:24 GMT

useACK = <boolean>
* Whether or not to use indexer acknowledgment.
* Indexer acknowledgment is an optional capability on forwarders that helps
  prevent loss of data when sending data to an indexer.

the workaround means you don't need use the indexer aknowledgment, so you run the risk of losing data during an indexer restart. The solution is not suitable for me.

Re: Why this error after upgrade to 9.0 "ERROR TcpOutputQ [ TcpOutEloop] - Unexpected event id=&a

hrawat — Thu, 08 Feb 2024 15:44:37 GMT

Upgrade to 9.0.3 and above.

Re: Why this error after upgrade to 9.0 "ERROR TcpOutputQ [ TcpOutEloop] - Unexpected event id=&a

pmerlin1 — Fri, 09 Feb 2024 07:50:35 GMT

The message appeared for forwarders (search head cluster) upgraded in 9.1.2

settings:

forceTimebasedAutoLB = false

useACK = true

autoLBFrequency = 30

Upgrading don't change the behavior on full enterprise splunk

But this seems to work on Universal Forwarder

Re: Why this error after upgrade to 9.0 "ERROR TcpOutputQ [ TcpOutEloop] - Unexpected event id=&a

hrawat — Fri, 09 Feb 2024 17:12:29 GMT

Can you let us know which log you see?

Following three logs

"Unexpected event id" ( 9.1.2 still logs)
"Invalid ACK received from indexer" ( 9.1.2 should not log)
"Got unexpected ACK with eventid" (9.1.2 should not log)

Re: Why this error after upgrade to 9.0 "ERROR TcpOutputQ [ TcpOutEloop] - Unexpected event id=&a

pmerlin1 — Mon, 12 Feb 2024 15:57:57 GMT

The message still logged is :

"Unexpected event id"

Re: Why this error after upgrade to 9.0 "ERROR TcpOutputQ [ TcpOutEloop] - Unexpected event id=&a

hrawat — Wed, 14 Feb 2024 13:34:42 GMT

This is expected and benign ERROR. We will change log to INFO in future.