Unable to export after upgrading to Splunk 9, hits "certificate is not valid for '127.0.0.1'"

[yuanliu]

Up to 8.5, I had no problem downloading results.  In 9.0.1, the server returns

<response>

  <messages>

    <msg type="ERROR">Service Unavailable</msg>

  </messages>

</response>

whenever I try to export (download), whether from search window or from dashboard.  The problem, it seems, is that 

/servicesNS/admin/search/search/jobs/<job id>/results/export invokes a python script that thinks my server_hostname is 127.0.0.1 (localhost) when the server's certification is for the server name. (I am using a publicly signed custom cert.)  Does anyone else get this problem?  How do you fix this?

To test, I run a simple search "| tstats count where index=_internal", then click the download/export button.  The server then returns the above error message.  web_service.log shows these errors:

2022-11-18 21:24:52,377 INFO	[6377f8245c7fc3f4089c10] startup:139 - Splunk appserver version=9.0.1 build=82c987350fde isFree=True isTrial=False
2022-11-18 21:24:52,415 ERROR	[6377f8245c7fc3f4089c10] __init__:868 - Socket error communicating with splunkd (error=[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: IP address mismatch, certificate is not valid for '127.0.0.1'. (_ssl.c:1106)), path = /servicesNS/admin/search/search/jobs/1668806682.36752/results/export?output_mode=csv&f=count&output_time_format=%25Y-%25m-%25dT%25H%3A%25M%3A%25S.%25Q%2B0000
2022-11-18 21:24:52,416 ERROR	[6377f8245c7fc3f4089c10] decorators:318 - Splunkd daemon is not responding: ("Error connecting to /servicesNS/admin/search/search/jobs/1668806682.36752/results/export?output_mode=csv&f=count&output_time_format=%25Y-%25m-%25dT%25H%3A%25M%3A%25S.%25Q%2B0000: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: IP address mismatch, certificate is not valid for '127.0.0.1'. (_ssl.c:1106)",)
Traceback (most recent call last):
  File "/opt/splunk/lib/python3.7/site-packages/splunk/rest/__init__.py", line 850, in streamingRequest
    conn.connect()
  File "/opt/splunk/lib/python3.7/http/client.py", line 1451, in connect
    server_hostname=server_hostname)
  File "/opt/splunk/lib/python3.7/ssl.py", line 428, in wrap_socket
    session=session
  File "/opt/splunk/lib/python3.7/ssl.py", line 878, in _create
    self.do_handshake()
  File "/opt/splunk/lib/python3.7/ssl.py", line 1147, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: IP address mismatch, certificate is not valid for '127.0.0.1'. (_ssl.c:1106)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/splunk/lib/python3.7/site-packages/splunk/appserver/mrsparkle/lib/decorators.py", line 304, in handle_exceptions
    return fn(self, *a, **kw)
  File "</opt/splunk/lib/python3.7/site-packages/decorator.py:decorator-gen-1471>", line 2, in getJobAsset
  File "/opt/splunk/lib/python3.7/site-packages/splunk/appserver/mrsparkle/lib/decorators.py", line 359, in apply_cache_headers
    response = fn(self, *a, **kw)
  File "/opt/splunk/lib/python3.7/site-packages/splunk/appserver/mrsparkle/controllers/search.py", line 392, in getJobAsset
    return self.streamJobExport(job, asset, **kwargs)
  File "/opt/splunk/lib/python3.7/site-packages/splunk/appserver/mrsparkle/controllers/search.py", line 154, in streamJobExport
    stream = rest.streamingRequest(uri, getargs=getargs, postargs=postargs, timeout=export_timeout)
  File "/opt/splunk/lib/python3.7/site-packages/splunk/rest/__init__.py", line 869, in streamingRequest
    raise splunk.SplunkdConnectionException('Error connecting to %s: %s' % (path, str(e)))
splunk.SplunkdConnectionException: Splunkd daemon is not responding: ("Error connecting to /servicesNS/admin/search/search/jobs/1668806682.36752/results/export?output_mode=csv&f=count&output_time_format=%25Y-%25m-%25dT%25H%3A%25M%3A%25S.%25Q%2B0000: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: IP address mismatch, certificate is not valid for '127.0.0.1'. (_ssl.c:1106)",)

Given no such problem before Splunk 9, I surmise that Splunk 9 added some server_name configuration for download that is not in my localization.  But I cannot find any.  My local configs are

server.conf - which contains serverName property that points to my server's name.

[general]
serverName = <my server domain>
pass4SymmKey = <some key>

[sslConfig]
sslVerifyServerCert = true
#cliVerifyServerName = true

# SSL settings
sslPassword = <some password>
serverCert = /var/opt/<some file path>.crt
caCertFile = /etc/pki/tls/certs/ca-bundle.crt

web.conf - contains no name, so I assume it uses serverName from server.conf

[settings]
httpport = 443
enableSplunkWebSSL = true

privKeyPath = /var/opt/<some file path>.key
serverCert = /var/opt/<some file path>.crt

[yuanliu · Accepted solution]

I knew Splunk Answers is going to give me answers😂  After verbalizing my analysis, I saw this line in a file named web.conf-old (which I saved while troubleshooting Dashboard Studio option not present in some 8.2.0 upgrades)

# location of splunkd; don't include http[s]:// in this anymore.
mgmtHostPort = 127.0.0.1:8089

So, I reviewed $SPLUNK_HOME/etc/system/default/web.conf.  This line is still here!  Clearly this didn't prevent export in 8.5 and earlier but Splunk 9 doesn't like it any more.  I added an override in local/web.conf so

[settings]
httpport = 443
mgmtHostPort = <my server name>:8089
enableSplunkWebSSL = true

privKeyPath = /var/opt/<some file path>.key
serverCert = /var/opt/<some file path>.crt

This "fixed" the problem.

I'll consider this "solved" for now.  But I still have an unanswered question: My installation is a standalone server, NOT universal forwarder.  According to web.conf.spec,

mgmtHostPort = <string>
* The host port of the splunkd process.
* The IP address and host port where Splunk Web looks for the splunkd process.
* The port listens on all avalible host IP addresses (0.0.0.0)
* Don't include "http[s]://" when specifying this setting. Only
  include the IP address and port.
* Default (on universal forwarders): localhost:8089
* Default (on all other Splunk platform instance types): 0.0.0.0:8089 

In other words, installer should have set 0.0.0.0:8089 in etc/system/default/web.conf .  Why is it 127.0.0.1:8089?