Getting Data In

Why this error after upgrade to 9.0 "ERROR TcpOutputQ [<thread id> TcpOutEloop] - Unexpected event id=<eventid>"?

hrawat_splunk
Splunk Employee
Splunk Employee

After upgrade to 9.0 seeing following

ERROR TcpOutputQ [<thread id> TcpOutEloop] - Unexpected event id=<eventid>

Tags (2)
0 Karma
1 Solution

hrawat_splunk
Splunk Employee
Splunk Employee

If useACK set to true and batch mode is on(default on) with Splunk 9.0, there is a possibility of hitting following error log messages.

"Unexpected event id"
"Invalid ACK received from indexer"
"Got unexpected ACK with eventid"

This may also lead to blocked queues on forwarding tier.

autoLBVolume and autoBatch while processing an event, apply limit using raw size of the event. However if there are  raw less events ( e.g. metrics events) autoLBVolume and autoBatch will end up sending lot more events then configured limits to receiver.
With autoLBVolume, it results in more than expected/configured events distributed to receivers.

With autoBatch, it results in batch of lot more events than expected. That means while a batch of thousands of events being sent to receiver, at the same time some events are already getting acknowledged.
Forwarder creates a list of  events to be acknowledged after successfully sending batch of events. However if the batch is in-flight over TCP layer and forwarder receives an ACKed event of the batch, it's not in the list of expected events to be acknowledged.  That leads to above ERROR.

Workaround: Either set useACK=false or autoBatch=false

Issue is fixed by 9.0.3 patch.

Note: 
After 9.0.3 upgrade, you will still see benign “Unexpected event id” log message. However there should not be following log messages.
"Invalid ACK received from indexer"
"Got unexpected ACK with eventid"


 

View solution in original post

vinayakwagh
Explorer

It helps Thanks

 

0 Karma

hrawat_splunk
Splunk Employee
Splunk Employee

If useACK set to true and batch mode is on(default on) with Splunk 9.0, there is a possibility of hitting following error log messages.

"Unexpected event id"
"Invalid ACK received from indexer"
"Got unexpected ACK with eventid"

This may also lead to blocked queues on forwarding tier.

autoLBVolume and autoBatch while processing an event, apply limit using raw size of the event. However if there are  raw less events ( e.g. metrics events) autoLBVolume and autoBatch will end up sending lot more events then configured limits to receiver.
With autoLBVolume, it results in more than expected/configured events distributed to receivers.

With autoBatch, it results in batch of lot more events than expected. That means while a batch of thousands of events being sent to receiver, at the same time some events are already getting acknowledged.
Forwarder creates a list of  events to be acknowledged after successfully sending batch of events. However if the batch is in-flight over TCP layer and forwarder receives an ACKed event of the batch, it's not in the list of expected events to be acknowledged.  That leads to above ERROR.

Workaround: Either set useACK=false or autoBatch=false

Issue is fixed by 9.0.3 patch.

Note: 
After 9.0.3 upgrade, you will still see benign “Unexpected event id” log message. However there should not be following log messages.
"Invalid ACK received from indexer"
"Got unexpected ACK with eventid"


 

anandhalagaras1
Communicator

@lawrence_magpoc ,

I am running with Splunk Universal Forwarder 9.0.2 in one of my Linux client machine and recently for the past couple of days i am getting this events in the internal logs and it seems like its getting crashed and once again the service is getting started automatically.

[build 17e00c557dc1] 2024-02-08 05:26:15 Received fatal signal 6 (Aborted) on PID 1908113. Cause: Signal sent by PID 1908113 running under UID 9991. Crashing thread: TcpOutEloop Registers: RIP: [0x00007F65EB39AACF] gsignal + 271 (libc.so.6 + 0x4EACF)

 

ERROR TcpOutputQ [1908232 TcpOutEloop] - Unexpected event id=30

ERROR TcpOutputQ [1908232 TcpOutEloop] - Unexpected event id=29

 

So how to fix this issue and also in which config file we need to add in the client machine where UF is running.

autoBatch=false

 

0 Karma

hrawat_splunk
Splunk Employee
Splunk Employee

In outputs.conf set

useACK=false

autoBatch=false
0 Karma

woodcock
Esteemed Legend

It is back in v9.0.1

0 Karma

hrawat_splunk
Splunk Employee
Splunk Employee

See my updated answer. 9.0.1 still logs the ERROR, but it does not block forwarder.

0 Karma

Sithima
Explorer

If the issue is fixed in 9.0.1, why am I getting the same error message in Splunk 9.0.1?

ERROR TcpOutputQ [<id> TcpOutEloop] - Unexpected eventid=<id>

0 Karma

hrawat_splunk
Splunk Employee
Splunk Employee

9.0.1 has not suppressed the ERROR log. It fixes the underlying tcpout queue blockage  issue. While you see the ERROR log but no tcpout queue blockage (as seen with 9.0.0) is an indication that the tcpout queue blockage  issue is resolved.

 

Will suppress 9.0.1 benign ERROR log in future releases. 

woodcock
Esteemed Legend

Actually the problem is still there.  I was getting continuous crashes on my HWF.

0 Karma

hrawat_splunk
Splunk Employee
Splunk Employee

That crash is still an issue and will be fixed.  It happens if forceTimebasedAutoLB=true
Workaround for 9.0.1 TcpOutputQ crash 
Set one of the following

forceTimebasedAutoLB=false

or

autoBatch=false

or

connectionsPerTarget=1

This crash is applicable if UF/HF resolves < 10 target IP addresses and forceTimebasedAutoLB=true.  

0 Karma

pmerlin1
Path Finder

The issue is not fixed after upgrading 9.1.2. This issue occured on search head cluster. 

My settings in outputs.conf : 

[indexer_discovery:target_master]
pass4SymmKey = **********

[tcpout]
defaultGroup = default_indexers
forceTimebasedAutoLB = true
maxQueueSize = 7MB
useACK = true

[tcpout:default_indexers]
server = **********01:9997,**********02.lan:9997



Tags (1)
0 Karma

hrawat_splunk
Splunk Employee
Splunk Employee

 

Following three logs

"Unexpected event id" ( 9.1.2 still logs)
"Invalid ACK received from indexer" ( 9.1.2 should not log)
"Got unexpected ACK with eventid" (9.1.2 should not log)


What exactly the issue you are hitting?

0 Karma

Vwagh
New Member

It needs to be 

 

UseAck = false

 

Then this errors should resolve.

0 Karma

pmerlin1
Path Finder
useACK = <boolean>
* Whether or not to use indexer acknowledgment.
* Indexer acknowledgment is an optional capability on forwarders that helps
  prevent loss of data when sending data to an indexer.

the workaround means you don't need use the indexer aknowledgment, so you run the risk of losing data during an indexer restart. The solution is not suitable for me.

0 Karma

hrawat_splunk
Splunk Employee
Splunk Employee

Upgrade to 9.0.3 and above.

0 Karma

pmerlin1
Path Finder

The message appeared for forwarders (search head cluster) upgraded in 9.1.2

settings: 

forceTimebasedAutoLB = false

useACK = true

autoLBFrequency = 30

Upgrading don't change the behavior on full enterprise splunk

 

But this seems to work on Universal Forwarder

0 Karma

hrawat_splunk
Splunk Employee
Splunk Employee

Can you let us know which log you see?


Following three logs

"Unexpected event id" ( 9.1.2 still logs)
"Invalid ACK received from indexer" ( 9.1.2 should not log)
"Got unexpected ACK with eventid" (9.1.2 should not log)

0 Karma

pmerlin1
Path Finder

The message still logged is : 

"Unexpected event id"

0 Karma

hrawat_splunk
Splunk Employee
Splunk Employee

This is expected and benign ERROR. We will change log to INFO in future.

0 Karma
Get Updates on the Splunk Community!

Stay Connected: Your Guide to November Tech Talks, Office Hours, and Webinars!

&#x1f342; Fall into November with a fresh lineup of Community Office Hours, Tech Talks, and Webinars we’ve ...

Transform your security operations with Splunk Enterprise Security

Hi Splunk Community, Splunk Platform has set a great foundation for your security operations. With the ...

Splunk Admins and App Developers | Earn a $35 gift card!

Splunk, in collaboration with ESG (Enterprise Strategy Group) by TechTarget, is excited to announce a ...