topic Re: Why is our Splunk 6.0.5 Universal Forwarder (HPUX) not contacting our Splunk 6.2.3 Deployment Server, even if connectivity exists. in Getting Data In

Why is our Splunk 6.0.5 Universal Forwarder (HPUX) not contacting our Splunk 6.2.3 Deployment Server, even if connectivity exists.

splunkn — Tue, 09 Jun 2015 13:23:38 GMT

I have installed Splunk Universal forwarder 6.0.5 in HPUX B.11.11 U 9000/800 box.

We are using deployment server (Splunk 6.2.3) to push apps.

But the HPUX box where we have installed splunk forwarder is not contacting our Deployment server.

While starting splunk for the first time during installation, we are getting the below message.

Splunk needs access to the system random number generator to generate
security certificates.  Normally this is provided by the /dev/urandom
device which is not present or accessible on this system. To fix this
problem, either:
  * download the "HP-UX Strong Number Generator" application package
    from HP's website
  * or, if the openssl package is installed on the system make sure the
    "prngd" daemon is running.  This is controlled at system startup
    by the /etc/rc.config.d/prngd file.

Do you want to continue anyway [y/n]? y

This appears to be your first time running this version of Splunk.

Splunk> See your world.  Maybe wish you hadn't................................

But splunk has started fine.

We are getting below error messages in splunkd.log

06-09-2015 12:15:32.504 +0100 INFO  DC:DeploymentClient - channel=tenantService/handshake Will retry sending handshake message to DS; err=not_connected
06-09-2015 12:15:34.360 +0100 ERROR HTTPClient - Should have gotten at least 3 tokens in status line, while getting response code.  Only got 0.
06-09-2015 12:15:34.360 +0100 INFO  HttpPubSubConnection - Secure HTTP POST failed: Unknown read error
06-09-2015 12:15:34.360 +0100 INFO  HttpPubSubConnection - Could not obtain connection, will retry after=78 seconds.
06-09-2015 12:15:44.524 +0100 INFO  DC:DeploymentClient - channel=tenantService/handshake Will retry sending handshake message to DS; err=not_connected

06-09-2015 12:11:37.125 +0100 ERROR ServerConfig - No '$SplunkHome/splunkforwarder/etc/auth/server.pem' certificate found.  Splunkd communication will not work without this!
06-09-2015 12:11:47.657 +0100 ERROR HTTPServer - SSL context could not be created - error in cert or password is wrong
06-09-2015 12:11:47.657 +0100 ERROR HTTPServer - SSL will not be enabled

Can anyone help us figure out why this Splunk universal forwarder is not contacting our Deployment server?

Re: Why is our Splunk 6.0.5 Universal Forwarder (HPUX) not contacting our Splunk 6.2.3 Deployment Server, even if connectivity exists.

splunker12er — Thu, 11 Jun 2015 06:41:21 GMT

Very good explanation & troubleshooting tips in this topic:
http://wiki.splunk.com/Community:Splunk2Splunk_SSL_DefaultCerts

Take a look at this error :

06-09-2015 12:11:37.125 +0100 ERROR ServerConfig - No '$SplunkHome/splunkforwarder/etc/auth/server.pem' certificate found.  Splunkd communication will not work without this!

in your deployment setup, your universal forwarder is not properly exchanged the keys for authentication for communication,

Copy the ''$SplunkHome/splunkforwarder/etc/auth/server.pem" from the deployment and paste in your deployment server under same path and restart.. then try again to reload the config from deployment to push the configs.

Re: Why is our Splunk 6.0.5 Universal Forwarder (HPUX) not contacting our Splunk 6.2.3 Deployment Server, even if connectivity exists.

splunkn — Thu, 11 Jun 2015 14:52:01 GMT

splunker12er,
Many thanks for your response, we have tried with copying ''$SplunkHome/splunk/etc/auth/server.pem from Deployment server to Universal forwarder, as in our case server.pem not found UF, instead of DS.
However no luck.

Do we need to do anything with enablesplunkdssl?
Because we have crossed errors related to splunkdssl certificates?