Forwarder splunklog extraction -
First time manual config for a Linux box. The server is set up to listen on 9997 and makes the connection but can't complete the data forwarding. Forwarder Management App on Server lists 0 forwarders have phoned home.
Any ideas?
11-24-2014 13:21:35.851 -0500 INFO DC:DeploymentClient - channel=tenantService/handshake Will retry sending handshake me
ssage to DS; err=not_connected
11-24-2014 13:21:36.903 -0500 WARN HttpPubSubConnection - Unable to parse message from PubSubSvr:
11-24-2014 13:21:36.903 -0500 INFO HttpPubSubConnection - Could not obtain connection, will retry after=37 seconds.
11-24-2014 13:21:47.689 -0500 INFO TcpOutputProc - Connected to idx=[MyServerIP]:9997
11-24-2014 13:21:47.851 -0500 INFO DC:DeploymentClient - channel=tenantService/handshake Will retry sending handshake me
ssage to DS; err=not_connected
11-24-2014 13:21:59.852 -0500 INFO DC:DeploymentClient - channel=tenantService/handshake Will retry sending handshake me
ssage to DS; err=not_connected
11-24-2014 13:22:11.852 -0500 INFO DC:DeploymentClient - channel=tenantService/handshake Will retry sending handshake me
ssage to DS; err=not_connected
11-24-2014 13:22:13.905 -0500 WARN HttpPubSubConnection - Unable to parse message from PubSubSvr:
OK, I'm answering my own question, which just makes me look dumb. But the Deployment client config got wired to the server data receiving port instead of the server management port.
This error message also occurs, when enable SSL on the forwarder, but have Non-SSL inputs (splunktcp://9997 instead of splunktcp-ssl:9997) on the indexer. Don't forget to add the SSL stanza in the inputs.conf on the indexer side.
http://wiki.splunk.com/Community:Splunk2Splunk_SSL_DefaultCerts
This should be the accepted answer.
OK, I'm answering my own question, which just makes me look dumb. But the Deployment client config got wired to the server data receiving port instead of the server management port.
don't feel dumb! I had the exact same problem and you helped me!
setup the deploy poll functionality:
splunk set deploy-poll <host>:<port>
ensure the port is the management port on the server (default is 8089) not the receiver listening port (default 9997). Check this in $SPLUNK_HOME/etc/system/local/deployment-client.conf.
forwarder still goes to 9997 (or whatever the port you have set the receiver to):
splunk add forward-server <host>:<port> -auth <username>:<password>
... provided by gethyn85, Problem in setting up forwarder and reciever ( Received unexpected 369295360 byte message)
I experience this issue too.
My current setup is:
1 Centos: NGINX Loadbalancer + Splunk Universal forwarder 6.5
Configurations locations $SPLUNK_HOME/etc/apps//local/
inputs.conf
[monitor:///var/log/nginx/acc*]
disabled = false
index = internal_loadbalancer
sourcetype = nginx:plus:access
whitelist = access.log(-|\.)\d*
ignoreOlderThan = 30h
[monitor:///var/log/nginx/e*]
disabled = false
index = internal_loadbalancer
sourcetype = nginx:plus:error
whitelist = error.log(-|\.)\d*
ignoreOlderThan = 30h
output.conf
[tcpout]
defaultGroup=indx
[tcpout:indx]
disabled=false
server=<indexIP>:9997
compressed=true
sendCookedData=ttue
autoLB=true
1 Centos: Splunk Enterprise 6.5
netstat -an | grep 9997
tcp 0 0 0.0.0.0:9997 0.0.0.0:* LISTEN
tcp 0 0 10.0.10.6:9997 10.0.10.5:56079 ESTABLISHED
Hey @drodman29
It doesn't make you look dumb. Answering and accepting your own answer after finding the solution helps other users on here that are coming across similar/identical issues. It's better than just leaving it open without a possible troubleshooting point, so thanks for resolving this post 🙂
Patrick
Additional info:
Server side splunkd.log has this:
11-24-2014 13:48:10.962 -0500 ERROR TcpInputProc - Message rejected. Received unexpected 369295616 byte message! from src={MyClientIP]:36189. Maximum message allowed: 67108864. (::)
I was getting the same error message when I tried to ingest data from CheckPoint LogExporter Log Server to Intermediate Heavy Forwarder.
I tried to use port numbers below 1024 and 9997.
It is not mentioned anywhere in the documentation that you can't use these ports to ingest data from LogExporter to Splunk.
It turned-out that you can't use port numbers below 1024 if you are not running as "root" or "root privileges". You can't also use port 9997, because it's reserved for "cooked" data ingestion from Splunk Forwarder to indexer or heavy forwarder.
In the end, I chose port 18188 and it worked.
I hope that this info helps someone who runs into the same problem as I did.