Monitoring Splunk

TcpInputProc - Received unexpected message

joonradley
Path Finder

This error keeps repeating in the error logs, but I have no idea what is causing it.

02-15-2011 14:55:31.161 ERROR TcpInputProc - Received unexpected 68021378 byte message! from hostname=tchuxxx.xxxx.com, ip=10.xx.xx.xx, port=50563

Is this related to the size of the message?

Thx

Tags (1)

jrodman
Splunk Employee
Splunk Employee

Essentially yes, it's saying that you got a big message. Since a 68MB data item is highly unlikely, there was probably some breakage in the datastream.

The protocol for splunk->splunk forwarding includes a length indicator number, which causes the receiving code to allocate memory. To avoid breaking the receiving Splunk, it does not blindly trust the size, but for cases of very large length numbers logs the problem and does not allocate the memory.

This could be a case where the forwarder is encountering some kind of memory corruption bug, where something is communicating to a splunktcp:// socket which is not quite conformant (hard to imagine, but possible), or when the stream of bytes in the tcp socket is getting messed up via some other means.

We had a known problem with early versions of 4.0.x and late versions of 3.4.x where the forwarder would sometimes inject 'heartbeat' pseudo-messages in the middle of other messages, corrupting the datastream. You may want to evaluate if tchuxxx.xxxx.com may be running an older version of splunk.

0 Karma

sf_user_199
Path Finder

Quick old-issue CPR...

We have this issue with a search head summarizing data & sending it back to our indexers. All the indexers are 5.0.2, as is the search head.

0 Karma

joonradley
Path Finder

The oldest version on the forwarders are 4.1.3.

0 Karma
Got questions? Get answers!

Join the Splunk Community Slack to learn, troubleshoot, and make connections with fellow Splunk practitioners in real time!

Meet up IRL or virtually!

Join Splunk User Groups to connect and learn in-person by region or remotely by topic or industry.

Get Updates on the Splunk Community!

Persistent Queue at TcpOut — One of Splunk's Most Practical Features

Splunk introduced persistent queueing at the tcpout layer as one of the most practical resilience features in ...

Skip the Awkward Silence: Have a .conf-ersation at .conf26

Picture this. You arrive at .conf26 already having your socializing and networking plans mapped out. No ...

Rethinking Zero Trust: From Product Purchases to Logical Control Evidence

Implementing Zero Trust (ZT) across complex environments often falters at the very beginning due to a ...