Why cliVerifyServerName failing because Splunk starts as 127.0.0.1?

[yaye]

Hi,

I want to run the command "splunk reload deploy-server" on my deployment server, but it fails with the following error:

[root@server etc]# su splunk
[splunk@server etc]$ splunk reload deploy-server
Your session is invalid.  Please login.
ERROR: IP address 127.0.0.1 not in server certificate. Please see server.conf/[sslConfig]/cliVerifyServerName for details.
Couldn't request server info: Couldn't complete HTTP request: error:14090086:SSL routines:ssl3_get_server_certificate:certificate verify failed

I'm running Splunk Enterprise 9.0.4.

The deployment server also acts as a license server and monitoring console.

Of course, my certificate does not have the localhost IP in it.

My Splunk has a Systemd Unit File.

#This unit file replaces the traditional start-up script for systemd
#configurations, and is used when enabling boot-start for Splunk on
#systemd-based Linux distributions.

[Unit]
Description=Systemd service file for Splunk, generated by 'splunk enable boot-start'
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
Restart=always
ExecStart=/data/splunk/bin/splunk _internal_launch_under_systemd
KillMode=mixed
KillSignal=SIGINT
TimeoutStopSec=360
LimitNOFILE=65536
LimitRTPRIO=99
SuccessExitStatus=51 52
RestartPreventExitStatus=51
RestartForceExitStatus=52
User=splunk
Group=splunk
Delegate=true
CPUShares=1024
MemoryLimit=24949776384
PermissionsStartOnly=true
ExecStartPost=-/bin/bash -c "chown -R splunk:splunk /sys/fs/cgroup/cpu/system.slice/%n"
ExecStartPost=-/bin/bash -c "chown -R splunk:splunk /sys/fs/cgroup/memory/system.slice/%n"

[Install]
WantedBy=multi-user.target

sslConfig Part of my server.conf

[sslConfig]
useClientSSLCompression = true
sslVersions = tls1.2
sslVerifyServerCert = true
sslVerifyServerName = true
requireClientCert = false
serverCert = <Combined PEM Cert>
sslRootCAPath = <Root CA PEM Cert>
sslPassword = <Password>
cliVerifyServerName = true

If you need any more info, let me know.

[isoutamo]

Hi

is this old environments (pre 9) which has upgraded?

Probably you have set mgmtHostPort to 127.0.0.1 on web.conf ? 

You should check it by 

 splunk btool web list settings --debug|egrep mgmtHostPort
$SPLUNK_HOME/etc/system/default/web.conf mgmtHostPort = 127.0.0.1:8089

If this is case, then try to remove is or bind this to real ip for FQDN which you have on your certificate.

mgmtHostPort = <string>
* The host port of the splunkd process.
* The IP address and host port where Splunk Web looks for the splunkd process.
* The port listens on all avalible host IP addresses (0.0.0.0)
* Don't include "http[s]://" when specifying this setting. Only
  include the IP address and port.
* Default (on universal forwarders): localhost:8089
* Default (on all other Splunk platform instance types): 0.0.0.0:8089

 r. Ismo

[yaye]

Yes, this is a upgraded environment.

[yaye]

We use DHCP for our servers, so it could be possible that after the lease time ends, the server has a new IP.

Is it possible to give mgmtHostPort a FQDN / CName?

[isoutamo]

If that FQDN is always resolvable, then it should work. It's just like localhost vs. 127.0.0.1 which (localhost) is currently default.

If this is server then static IP would be better (IMHO). Of course if it's e.g. in AWS then you just need to update r53 automatically to keep ip vs FQDN in sync. Other option is add secondary interface with static ip.

[yaye]

Changed the mgmtHostPort to the FQDN of the server.

This FQDN stands in the Certificate Subject Alternative Name of the certificate of the server.

[settings]

startwebserver = 1

mgmtHostPort = <redacted>:8089

enableSplunkWebSSL = true

privKeyPath = <redacted>

serverCert = <redacted>

sslVersions = tls1.2

max_upload_size = 2048

Changed cliVerifyServerName to true

[sslConfig]

useClientSSLCompression = true

sslVersions = tls1.2

sslVerifyServerCert = true

sslVerifyServerName = true

requireClientCert = false

serverCert = <redacted>

sslRootCAPath = <redacted>

sslPassword = <redacted>

cliVerifyServerName = true

Getting the same error after restarting the server

[root@<redacted> ~]# systemctl restart Splunkd

[root@<redacted> ~]# su splunk

[splunk@<redacted> root]$ splunk reload deploy-server

Your session is invalid.  Please login.

ERROR: IP address 127.0.0.1 not in server certificate. Please see server.conf/[sslConfig]/cliVerifyServerName for details.

Couldn't request server info: Couldn't complete HTTP request: error:14090086:SSL routines:ssl3_get_server_certificate:certificate verify failed

[FlorianScho]

You got that fixed?

[yaye]

No, unfortunately I didn't get it solved.

But I didn't spend any more time on the problem.