Did the SSL certificate requirement for management port 8089 change with Splunk version 6.3.2?

[rgsurfs]

I have 20 Windows servers and 7 Linux servers. Each server has a unique DOD server certificate assigned to the FQDN and alternate name Shortname.

When I would go to the browser to check the status: https://FQDN:8089 the web page would be displayed, with the DOD certificate.

Now, after I upgraded the forwarders to 6.3.2 every server (Windows and Linux) only shows the default Splunk cert when I access the webpage.

My web.conf file did not change, my certs did not change.

No matter what I check / try, I cannot get the management port to use the DOD Cert anymore? Did something change with v6.3.2 and later?

[rgsurfs]

I usually upgraded the Splunk servers and then forwarders. Oh well. I've since moved out of the Splunk area and gave it to somebody else !!!!

[pil321]

rgsurfs,

I'm a little late to the party - I stumbled across this looking for 'DOD cert' answers.

When you upgraded your forwarders to 6.3.2, was Splunk enterprise also updated to that version first?

I remember a conversation a long time ago with a Splunk engineer: make sure that your Splunk forwarder version is <= to the Splunk enterprise version. Otherwise....bad juju.

[dwaddle]

Were you using the default file and directory names in $SPLUNK_HOME/etc/auth ? The upgrade / migration scripts starting in 6.3 would auto-renew a default self-signed cert if it was close to expiration. My first thought is that if you re-used the same files, the upgrader could have clobbered yours with new certs. But, this is just a guess given limited information.

[rgsurfs]

I ran ./splunk cmd btool web list My five stanzas above, listed under [settings], were returned.

So, I also tested web.conf by changing caCertPath to cacertpath and restarting splunk, Splunk restarted and displayed an error about cacpertpath being invalid. So I know Splunk is looking at my web.conf.

It seems Splunk is just ignoring my certs and using it's default no matter what I do.

[rgsurfs]

I did ./splunk btool check --debug | find "web.conf" returned four:

d:\splunkuniversalforwarder\etc\apps\splunkuniversalforwarder\default\web.conf
d:\splunkuniversalforwarder\etc\apps\splunk_TA_Windows\default\web.conf
d:\splunkuniversalforwarder\etc\system\default\web.conf
d:\splunkuniversalforwarder\etc\system\local\web.conf

[rgsurfs]

I did ./splunk btool web list --debug | find "web.conf"

d:\splunkuniversalforwarder\etc\apps\splunkuniversalforwarder\default\web.conf [settings]
d:\splunkuniversalforwarder\etc\apps\splunkuniversalforwarder\default\web.conf startwebserver=0
d:\splunkuniversalforwarder\etc\system\local\web.conf enableSplunkWebSSL = true
d:\splunkuniversalforwarder\etc\system\local\web.conf privKeyPath = etc/auth/DOD/web_private.pem
d:\splunkuniversalforwarder\etc\system\local\web.conf caCertPath = /etc/auth/DOD/web_chain.pem
d:\splunkuniversalforwarder\etc\system\local\web.conf sslVersions = tls1.1, tls1.2
d:\splunkuniversalforwarder\etc\system\local\web.conf cipherSuite = high

[rgsurfs]

In version 6.3.0 all of this works:
My web.conf file:
[settings]
enableSplunkWebSSL = true
privKeyPath = etc/auth/DOD/web_private.pem
caCertPath = /etc/auth/DOD/web_chain.pem
sslVersions = tls1.1, tls1.2
cipherSuite = high

web_private.pem is my private key, without password

web_chain.pem is the chain key:

server cert
subordinate cert
intermediate cert
root CA cert

but in 6.3.2, does not work anymore.

[dwaddle]

Okay, I'm stumped then. Do you know how to use btool to validate configurations? I would use btool to make sure the configs in your web.conf are not being ignored / overlayed by some other web.conf elsewhere in the config structure.

[rgsurfs]

I just went back and found a server that I did not upgrade the forwarder on. This server is using splunkforwarder v6.3.0 and the web page is configured and displays correctly with the DOD certificate.

I compared the web.conf and they are the same on both servers.

It looks like the upgrade to 6.3.2 does something and now I have issues with all of my web pages that are on port 8089

[rgsurfs]

So, today, I went back to this 6.3.0 server. I have removed the web.conf completely from etc/system/local/
restarted splunk, and yet the web page still comes up!!! with the DOD SSL Cert assigned.

I cleared cache on my browsers, closed and reopened browser, and yet there it is again, the web page, running under 6.3.0, with no web.conf file, yet using personalized DOD certs ???????????

[rgsurfs]

I went to the server with v6.3.0, it's RHEL6.7. I did a rpm -Uhv splunkforwarder6.3.2...x86_64.rpm

it fails on the upgrade and ends with /splunk migrate renew-cert failure

I uninstalled (rpm -e) the splunkforwarder v6.3.0 rpm and tried installing splunkforwarder v6.3.2 (rpm -ihv), again failed with the migrate failure.

I uninstalled the forwarder again, and searched the server and deleted all splunk files/folders.

I now successfully installed v6.3.2. However, it's only using the default certificates for 8089. I copied the old (6.3.0) configs over and still no joy. there's something about v6.3.2 that does not like DOD certs or the web.conf flie.