Monitoring Splunk

TcpInputProc - Received unexpected message

joonradley
Path Finder

This error keeps repeating in the error logs, but I have no idea what is causing it.

02-15-2011 14:55:31.161 ERROR TcpInputProc - Received unexpected 68021378 byte message! from hostname=tchuxxx.xxxx.com, ip=10.xx.xx.xx, port=50563

Is this related to the size of the message?

Thx

Tags (1)

jrodman
Splunk Employee
Splunk Employee

Essentially yes, it's saying that you got a big message. Since a 68MB data item is highly unlikely, there was probably some breakage in the datastream.

The protocol for splunk->splunk forwarding includes a length indicator number, which causes the receiving code to allocate memory. To avoid breaking the receiving Splunk, it does not blindly trust the size, but for cases of very large length numbers logs the problem and does not allocate the memory.

This could be a case where the forwarder is encountering some kind of memory corruption bug, where something is communicating to a splunktcp:// socket which is not quite conformant (hard to imagine, but possible), or when the stream of bytes in the tcp socket is getting messed up via some other means.

We had a known problem with early versions of 4.0.x and late versions of 3.4.x where the forwarder would sometimes inject 'heartbeat' pseudo-messages in the middle of other messages, corrupting the datastream. You may want to evaluate if tchuxxx.xxxx.com may be running an older version of splunk.

0 Karma

sf_user_199
Path Finder

Quick old-issue CPR...

We have this issue with a search head summarizing data & sending it back to our indexers. All the indexers are 5.0.2, as is the search head.

0 Karma

joonradley
Path Finder

The oldest version on the forwarders are 4.1.3.

0 Karma
Get Updates on the Splunk Community!

Welcome to the Splunk Community!

(view in My Videos) We're so glad you're here! The Splunk Community is place to connect, learn, give back, and ...

Tech Talk | Elevating Digital Service Excellence: The Synergy of Splunk RUM & APM

Elevating Digital Service Excellence: The Synergy of Real User Monitoring and Application Performance ...

Adoption of RUM and APM at Splunk

    Unleash the power of Splunk Observability   Watch Now In this can't miss Tech Talk! The Splunk Growth ...