I am installing the universal forwarder (6.2) on redhat. I am running into several issues with the SSL setup. I am using my own selfsigned certs. This is working fine in an old 4.2 universal forwarder setup.
After extracting splunk I do the following:
1) Copy my certs to /etc/auth/server.pem and /etc/auth/ca.pem
2) update /etc/system/local/inputs.conf with
... [tcpout-server://splunkserver.ec2.local:9997] sslCertPath = /usr/local/splunkforwarder/etc/auth/server.pem sslPassword = mypassword sslRootCAPath = /usr/local/splunkforwarder/etc/auth/ca.pem sslVerifyServerCert = false ...
3) Update etc/system/default/server.conf
... sslPassword = mypassword ...
4) Start splunk server with no configuration errors and etc/system/local/server.conf is generated
5) Find this error in splunkd.log
08-04-2016 13:07:13.134 -0700 ERROR TcpOutputFd - Connection to host=x.x.x.x:9997 failed. sock_error = 0. SSL Error = error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed
What am I missing? Do I need to care about splunk.secret that manages the encryption of the sslpassword value in /etc/sytem/local/outputs and /etc/system/local/server.conf?
I can open my cert with password when doing:
openssl rsa -in /usr/local/splunkforwarder/etc/auth/server.pem -text
So the cert and passphrase is correct. What else should I consider? I have stopped splunk and set the sslKeysfilePassword in etc/system/local/server.conf. Start splunk but no luck. I have also tried the same for the sslPassword in etc/system/local/outputs.conf but not luck.
Any advice would be appreciated
A leadoff comment - do not make changes to things in
$SPLUNK_HOME/etc/system/default -- or in any other app-level default unless you are the author of the app. Changes made in default will be overwritten without warning during upgrades, and making the right change in a local file will override the defaults anyway.
Splunk uses different SSL settings in different configuration files for each type of connection. I'm not sure if your notes above are correct or not, because you've got a
[tcpout-server] stanza listed in
inputs.conf (it should be in
outputs.conf), and you're updating a password in
etc/system/default/server.conf that is most undoubtedly being overlaid by
By self-signed do you mean actually self-signed, or do you mean signed by your private certificate authority? These are two different things. Self-signed is like a tautology -- I am who I say I am because I say that is who I am. A private CA has a root certificate (which is probably self-signed) that establishes a trust anchor for other certificates that the CA signs.
I would use btool to look at how the various SSL settings are configured. Something like:
splunk cmd btool --debug outputs list
Might help you discover a configuration file somewhere that does have a setting like
sslVerifyServerCert = true, which is what the original error you reported sounds like. Situational awareness of where settings are done, and how different configuration files overlay, is of absolute importance when trying to understand why Splunk is doing something. The docs talk about this at http://docs.splunk.com/Documentation/Splunk/6.4.2/Admin/Wheretofindtheconfigurationfiles
Finally, (plugging mine and @starcher's work some) you should have a look at this .conf talk material:
This is a whole cookbook intended to help you configure all the SSL things in Splunk correctly.
You are correct I mistakenly listed inputs.conf instead of outputs.conf in step 2 in the initial question.
I made a change to $SPLUNKHOME/etc/system/default/server.conf knowing that it is probably not a good idea but how else can I change the default sslPassword? Seems to be propagated when I start the server to $SPLUNKHOME/etc/system/local/server.conf (which is not there on first startup). I should probably start the app up the first time then ensure I overwrite the correct sslpassword to $SPLUNK_HOME/etc/system/local/server.conf after?
By self signed I mean that I created it by following the splunk community wiki instructions for self signed cert with new root CA. http://wiki.splunk.com/Community:Splunk2Splunk_SSL_SelfSignedCert_NewRootCA
I used the splunk btool before to check configuration values but did not explicitly look for sslVerifyServerCert = true. Thanks for the tip, I will look out for that.
Thanks for the links. I will educate myself before I dig into the problem again
Did you ever figure this out wouterr? I'm hitting the same problem in 6.4
I did not solve this problem. Instead of attempting to fix this we decided to blow away the old install and re-install from scratch. That seemed to have resolved this issue for some reason. Sorry not very helpful if you are in a situation where you cant afford to do that.
That's unfortunate but thanks for letting me know. A reinstall really isn't possible at this point. I'll keep digging. I'll update you if I do find a solution though.