Getting Data In

6.2 Forwarder Configuration on Linux: Why am I getting error "TcpInputProc - Message rejected. Received unexpected 369295616 byte message!" in server's splunkd.log?

Path Finder

Forwarder splunklog extraction -
First time manual config for a Linux box. The server is set up to listen on 9997 and makes the connection but can't complete the data forwarding. Forwarder Management App on Server lists 0 forwarders have phoned home.

Any ideas?

11-24-2014 13:21:35.851 -0500 INFO  DC:DeploymentClient - channel=tenantService/handshake Will retry sending handshake me
ssage to DS; err=not_connected
11-24-2014 13:21:36.903 -0500 WARN  HttpPubSubConnection - Unable to parse message from PubSubSvr:
11-24-2014 13:21:36.903 -0500 INFO  HttpPubSubConnection - Could not obtain connection, will retry after=37 seconds.
11-24-2014 13:21:47.689 -0500 INFO  TcpOutputProc - Connected to idx=[MyServerIP]:9997
11-24-2014 13:21:47.851 -0500 INFO  DC:DeploymentClient - channel=tenantService/handshake Will retry sending handshake me
ssage to DS; err=not_connected
11-24-2014 13:21:59.852 -0500 INFO  DC:DeploymentClient - channel=tenantService/handshake Will retry sending handshake me
ssage to DS; err=not_connected
11-24-2014 13:22:11.852 -0500 INFO  DC:DeploymentClient - channel=tenantService/handshake Will retry sending handshake me
ssage to DS; err=not_connected
11-24-2014 13:22:13.905 -0500 WARN  HttpPubSubConnection - Unable to parse message from PubSubSvr:
0 Karma
1 Solution

Path Finder

OK, I'm answering my own question, which just makes me look dumb. But the Deployment client config got wired to the server data receiving port instead of the server management port.

View solution in original post

Communicator

This error message also occurs, when enable SSL on the forwarder, but have Non-SSL inputs (splunktcp://9997 instead of splunktcp-ssl:9997) on the indexer. Don't forget to add the SSL stanza in the inputs.conf on the indexer side.

http://wiki.splunk.com/Community:Splunk2Splunk_SSL_DefaultCerts

Esteemed Legend

This should be the accepted answer.

0 Karma

Path Finder

OK, I'm answering my own question, which just makes me look dumb. But the Deployment client config got wired to the server data receiving port instead of the server management port.

View solution in original post

Engager

don't feel dumb! I had the exact same problem and you helped me!

Explorer

setup the deploy poll functionality:

splunk set deploy-poll <host>:<port>

ensure the port is the management port on the server (default is 8089) not the receiver listening port (default 9997). Check this in $SPLUNK_HOME/etc/system/local/deployment-client.conf.

forwarder still goes to 9997 (or whatever the port you have set the receiver to):

splunk add forward-server <host>:<port> -auth <username>:<password>

... provided by gethyn85, Problem in setting up forwarder and reciever ( Received unexpected 369295360 byte message)

New Member

I experience this issue too.

My current setup is:
1 Centos: NGINX Loadbalancer + Splunk Universal forwarder 6.5
Configurations locations $SPLUNK_HOME/etc/apps//local/
inputs.conf

[monitor:///var/log/nginx/acc*]
disabled = false
index = internal_loadbalancer
sourcetype = nginx:plus:access
whitelist = access.log(-|\.)\d*
ignoreOlderThan = 30h


[monitor:///var/log/nginx/e*]
disabled = false
index = internal_loadbalancer
sourcetype = nginx:plus:error
whitelist = error.log(-|\.)\d*
ignoreOlderThan = 30h

output.conf

[tcpout]
defaultGroup=indx

[tcpout:indx]
disabled=false
server=<indexIP>:9997  
compressed=true
sendCookedData=ttue
autoLB=true

1 Centos: Splunk Enterprise 6.5

netstat -an | grep 9997
tcp        0      0 0.0.0.0:9997                0.0.0.0:*                   LISTEN      
tcp        0      0 10.0.10.6:9997              10.0.10.5:56079             ESTABLISHED 
0 Karma

Community Manager
Community Manager

Hey @drodman29

It doesn't make you look dumb. Answering and accepting your own answer after finding the solution helps other users on here that are coming across similar/identical issues. It's better than just leaving it open without a possible troubleshooting point, so thanks for resolving this post 🙂

Patrick

Path Finder

Additional info:
Server side splunkd.log has this:
11-24-2014 13:48:10.962 -0500 ERROR TcpInputProc - Message rejected. Received unexpected 369295616 byte message! from src={MyClientIP]:36189. Maximum message allowed: 67108864. (::)

0 Karma