Why am I unable to forward data from a Splunk forwarder to Splunk Cloud on Windows?

[jgorman_THG]

Hello,

I have been trying for the last 8 hours to forward data to a Splunk Cloud instance. I generated the credentials off the Splunk Cloud instance as directed and attempted to use them on a heavy forwarder to no avail.

I also tried a universal forwarder as well but it just won't work. I believe the problem is related to the credentials.

One particular message I received was:

12-02-2016 19:27:20.156 -0500 WARN TcpOutputProc - 'sslCertPath' deprecated; use 'clientCert' instead 
12-02-2016 19:27:20.156 -0500 WARN TcpOutputProc - 'sslCertPath' deprecated; use 'clientCert' instead 
12-02-2016 19:27:20.156 -0500 WARN TcpOutputProc - 'sslCertPath' deprecated; use 'clientCert' instead 
12-02-2016 19:27:20.156 -0500 WARN TcpOutputProc - 'sslCertPath' deprecated; use 'clientCert' instead 
12-02-2016 19:27:20.156 -0500 WARN TcpOutputProc - 'sslCertPath' deprecated; use 'clientCert' instead 

I made a change to the config files to fix this, but it still will not work.

In splunkd.log all I see is:

12-02-2016 19:38:59.726 -0500 INFO  DC:DeploymentClient - channel=tenantService/handshake Will retry sending handshake message to DS; err=not_connected
12-02-2016 19:39:07.772 -0500 WARN  TcpOutputProc - Cooked connection to ip=52.55.109.251:9997 timed out
12-02-2016 19:39:11.737 -0500 INFO  DC:DeploymentClient - channel=tenantService/handshake Will retry sending handshake message to DS; err=not_connected
12-02-2016 19:39:23.739 -0500 INFO  DC:DeploymentClient - channel=tenantService/handshake Will retry sending handshake message to DS; err=not_connected
12-02-2016 19:39:27.664 -0500 WARN  TcpOutputProc - Cooked connection to ip=52.204.196.213:9997 timed out
12-02-2016 19:39:35.740 -0500 INFO  DC:DeploymentClient - channel=tenantService/handshake Will retry sending handshake message to DS; err=not_connected
12-02-2016 19:39:44.356 -0500 WARN  HttpPubSubConnection - Unable to parse message from PubSubSvr: 
12-02-2016 19:39:44.356 -0500 INFO  HttpPubSubConnection - Could not obtain connection, will retry after=84.982 seconds.
12-02-2016 19:39:47.553 -0500 WARN  TcpOutputProc - Cooked connection to ip=52.44.41.196:9997 timed out
12-02-2016 19:39:47.740 -0500 INFO  DC:DeploymentClient - channel=tenantService/handshake Will retry sending handshake message to DS; err=not_connected

Any ideas?

Thanks,

JG

[goodsellt]

Have you done any network diagnostics from that box to the Splunk cloud endpoint? Make sure the ports Splunk cloud is asking you to use for data transmission are working correctly.

After that verify everything in the SSL config is as they say it should be, and if there is a password for the cert file, put it in plaintext back in the config and reboot the box so it can be resalted.

I've experienced similar issues before and it was because the SSL config was not perfect (however I'm on an on-prem deployment), you should start with network diagnostics then move onto triple checking the SSL config.

[gneumann_splunk]

I can't give any input about the messages you are receiving, but try reviewing these topics to confirm you have configured your forwarders and credentials correctly.
http://docs.splunk.com/Documentation/Forwarder/6.5.1/Forwarder/HowtoforwarddatatoSplunkCloud
http://docs.splunk.com/Documentation/SplunkCloud/6.5.1/User/ForwardDataToSplunkCloudFromWindows