Monitoring Splunk

TcpInputProc - Received unexpected message

joonradley
Path Finder

This error keeps repeating in the error logs, but I have no idea what is causing it.

02-15-2011 14:55:31.161 ERROR TcpInputProc - Received unexpected 68021378 byte message! from hostname=tchuxxx.xxxx.com, ip=10.xx.xx.xx, port=50563

Is this related to the size of the message?

Thx

Tags (1)

jrodman
Splunk Employee
Splunk Employee

Essentially yes, it's saying that you got a big message. Since a 68MB data item is highly unlikely, there was probably some breakage in the datastream.

The protocol for splunk->splunk forwarding includes a length indicator number, which causes the receiving code to allocate memory. To avoid breaking the receiving Splunk, it does not blindly trust the size, but for cases of very large length numbers logs the problem and does not allocate the memory.

This could be a case where the forwarder is encountering some kind of memory corruption bug, where something is communicating to a splunktcp:// socket which is not quite conformant (hard to imagine, but possible), or when the stream of bytes in the tcp socket is getting messed up via some other means.

We had a known problem with early versions of 4.0.x and late versions of 3.4.x where the forwarder would sometimes inject 'heartbeat' pseudo-messages in the middle of other messages, corrupting the datastream. You may want to evaluate if tchuxxx.xxxx.com may be running an older version of splunk.

0 Karma

sf_user_199
Path Finder

Quick old-issue CPR...

We have this issue with a search head summarizing data & sending it back to our indexers. All the indexers are 5.0.2, as is the search head.

0 Karma

joonradley
Path Finder

The oldest version on the forwarders are 4.1.3.

0 Karma
Got questions? Get answers!

Join the Splunk Community Slack to learn, troubleshoot, and make connections with fellow Splunk practitioners in real time!

Meet up IRL or virtually!

Join Splunk User Groups to connect and learn in-person by region or remotely by topic or industry.

Get Updates on the Splunk Community!

Why Splunk Customers Should Attend Cisco Live 2026 Las Vegas

Why Splunk Customers Should Attend Cisco Live 2026 Las Vegas     Cisco Live 2026 is almost here, and this ...

What Is the Name of the USB Key Inserted by Bob Smith? (BOTS Hint, Not the Answer)

Hello Splunkers,   So you searched, “what is the name of the usb key inserted by bob smith?”  Not gonna lie… ...

Automating Threat Operations and Threat Hunting with Recorded Future

    Automating Threat Operations and Threat Hunting with Recorded Future June 29, 2026 | Register   Is your ...