Monitoring Splunk

TcpInputProc - Received unexpected message

joonradley
Path Finder

This error keeps repeating in the error logs, but I have no idea what is causing it.

02-15-2011 14:55:31.161 ERROR TcpInputProc - Received unexpected 68021378 byte message! from hostname=tchuxxx.xxxx.com, ip=10.xx.xx.xx, port=50563

Is this related to the size of the message?

Thx

Tags (1)

jrodman
Splunk Employee
Splunk Employee

Essentially yes, it's saying that you got a big message. Since a 68MB data item is highly unlikely, there was probably some breakage in the datastream.

The protocol for splunk->splunk forwarding includes a length indicator number, which causes the receiving code to allocate memory. To avoid breaking the receiving Splunk, it does not blindly trust the size, but for cases of very large length numbers logs the problem and does not allocate the memory.

This could be a case where the forwarder is encountering some kind of memory corruption bug, where something is communicating to a splunktcp:// socket which is not quite conformant (hard to imagine, but possible), or when the stream of bytes in the tcp socket is getting messed up via some other means.

We had a known problem with early versions of 4.0.x and late versions of 3.4.x where the forwarder would sometimes inject 'heartbeat' pseudo-messages in the middle of other messages, corrupting the datastream. You may want to evaluate if tchuxxx.xxxx.com may be running an older version of splunk.

0 Karma

sf_user_199
Path Finder

Quick old-issue CPR...

We have this issue with a search head summarizing data & sending it back to our indexers. All the indexers are 5.0.2, as is the search head.

0 Karma

joonradley
Path Finder

The oldest version on the forwarders are 4.1.3.

0 Karma
Get Updates on the Splunk Community!

Dashboards: Hiding charts while search is being executed and other uses for tokens

There are a couple of features of SimpleXML / Classic dashboards that can be used to enhance the user ...

Splunk Observability Cloud's AI Assistant in Action Series: Explaining Metrics and ...

This is the fourth post in the Splunk Observability Cloud’s AI Assistant in Action series that digs into how ...

Brains, Bytes, and Boston: Learn from the Best at .conf25

When you think of Boston, you might picture colonial charm, world-class universities, or even the crack of a ...