Installation

Unable to export after upgrading to Splunk 9, hits "certificate is not valid for '127.0.0.1'"

yuanliu
SplunkTrust
SplunkTrust

Up to 8.5, I had no problem downloading results.  In 9.0.1, the server returns

<response>
  <messages>
    <msg type="ERROR">Service Unavailable</msg>
  </messages>
</response>
whenever I try to export (download), whether from search window or from dashboard.  The problem, it seems, is that 

/servicesNS/admin/search/search/jobs/<job id>/results/export invokes a python script that thinks my server_hostname is 127.0.0.1 (localhost) when the server's certification is for the server name. (I am using a publicly signed custom cert.)  Does anyone else get this problem?  How do you fix this?

To test, I run a simple search "| tstats count where index=_internal", then click the download/export button.  The server then returns the above error message.  web_service.log shows these errors:

 

2022-11-18 21:24:52,377 INFO	[6377f8245c7fc3f4089c10] startup:139 - Splunk appserver version=9.0.1 build=82c987350fde isFree=True isTrial=False
2022-11-18 21:24:52,415 ERROR	[6377f8245c7fc3f4089c10] __init__:868 - Socket error communicating with splunkd (error=[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: IP address mismatch, certificate is not valid for '127.0.0.1'. (_ssl.c:1106)), path = /servicesNS/admin/search/search/jobs/1668806682.36752/results/export?output_mode=csv&f=count&output_time_format=%25Y-%25m-%25dT%25H%3A%25M%3A%25S.%25Q%2B0000
2022-11-18 21:24:52,416 ERROR	[6377f8245c7fc3f4089c10] decorators:318 - Splunkd daemon is not responding: ("Error connecting to /servicesNS/admin/search/search/jobs/1668806682.36752/results/export?output_mode=csv&f=count&output_time_format=%25Y-%25m-%25dT%25H%3A%25M%3A%25S.%25Q%2B0000: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: IP address mismatch, certificate is not valid for '127.0.0.1'. (_ssl.c:1106)",)
Traceback (most recent call last):
  File "/opt/splunk/lib/python3.7/site-packages/splunk/rest/__init__.py", line 850, in streamingRequest
    conn.connect()
  File "/opt/splunk/lib/python3.7/http/client.py", line 1451, in connect
    server_hostname=server_hostname)
  File "/opt/splunk/lib/python3.7/ssl.py", line 428, in wrap_socket
    session=session
  File "/opt/splunk/lib/python3.7/ssl.py", line 878, in _create
    self.do_handshake()
  File "/opt/splunk/lib/python3.7/ssl.py", line 1147, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: IP address mismatch, certificate is not valid for '127.0.0.1'. (_ssl.c:1106)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/splunk/lib/python3.7/site-packages/splunk/appserver/mrsparkle/lib/decorators.py", line 304, in handle_exceptions
    return fn(self, *a, **kw)
  File "</opt/splunk/lib/python3.7/site-packages/decorator.py:decorator-gen-1471>", line 2, in getJobAsset
  File "/opt/splunk/lib/python3.7/site-packages/splunk/appserver/mrsparkle/lib/decorators.py", line 359, in apply_cache_headers
    response = fn(self, *a, **kw)
  File "/opt/splunk/lib/python3.7/site-packages/splunk/appserver/mrsparkle/controllers/search.py", line 392, in getJobAsset
    return self.streamJobExport(job, asset, **kwargs)
  File "/opt/splunk/lib/python3.7/site-packages/splunk/appserver/mrsparkle/controllers/search.py", line 154, in streamJobExport
    stream = rest.streamingRequest(uri, getargs=getargs, postargs=postargs, timeout=export_timeout)
  File "/opt/splunk/lib/python3.7/site-packages/splunk/rest/__init__.py", line 869, in streamingRequest
    raise splunk.SplunkdConnectionException('Error connecting to %s: %s' % (path, str(e)))
splunk.SplunkdConnectionException: Splunkd daemon is not responding: ("Error connecting to /servicesNS/admin/search/search/jobs/1668806682.36752/results/export?output_mode=csv&f=count&output_time_format=%25Y-%25m-%25dT%25H%3A%25M%3A%25S.%25Q%2B0000: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: IP address mismatch, certificate is not valid for '127.0.0.1'. (_ssl.c:1106)",)

 

Given no such problem before Splunk 9, I surmise that Splunk 9 added some server_name configuration for download that is not in my localization.  But I cannot find any.  My local configs are

server.conf - which contains serverName property that points to my server's name.

 

[general]
serverName = <my server domain>
pass4SymmKey = <some key>

[sslConfig]
sslVerifyServerCert = true
#cliVerifyServerName = true

# SSL settings
sslPassword = <some password>
serverCert = /var/opt/<some file path>.crt
caCertFile = /etc/pki/tls/certs/ca-bundle.crt

 

web.conf - contains no name, so I assume it uses serverName from server.conf

 

[settings]
httpport = 443
enableSplunkWebSSL = true

privKeyPath = /var/opt/<some file path>.key
serverCert = /var/opt/<some file path>.crt

 

Labels (2)
0 Karma
1 Solution

yuanliu
SplunkTrust
SplunkTrust

I knew Splunk Answers is going to give me answers😂  After verbalizing my analysis, I saw this line in a file named web.conf-old (which I saved while troubleshooting Dashboard Studio option not present in some 8.2.0 upgrades)

 

# location of splunkd; don't include http[s]:// in this anymore.
mgmtHostPort = 127.0.0.1:8089

 

So, I reviewed $SPLUNK_HOME/etc/system/default/web.conf.  This line is still here!  Clearly this didn't prevent export in 8.5 and earlier but Splunk 9 doesn't like it any more.  I added an override in local/web.conf so

 

[settings]
httpport = 443
mgmtHostPort = <my server name>:8089
enableSplunkWebSSL = true

privKeyPath = /var/opt/<some file path>.key
serverCert = /var/opt/<some file path>.crt

 

This "fixed" the problem.

I'll consider this "solved" for now.  But I still have an unanswered question: My installation is a standalone server, NOT universal forwarder.  According to web.conf.spec,

mgmtHostPort = <string>
* The host port of the splunkd process.
* The IP address and host port where Splunk Web looks for the splunkd process.
* The port listens on all avalible host IP addresses (0.0.0.0)
* Don't include "http[s]://" when specifying this setting. Only
  include the IP address and port.
* Default (on universal forwarders): localhost:8089
* Default (on all other Splunk platform instance types): 0.0.0.0:8089 

In other words, installer should have set 0.0.0.0:8089 in etc/system/default/web.conf .  Why is it 127.0.0.1:8089?

View solution in original post

Tags (1)
0 Karma

yuanliu
SplunkTrust
SplunkTrust

I knew Splunk Answers is going to give me answers😂  After verbalizing my analysis, I saw this line in a file named web.conf-old (which I saved while troubleshooting Dashboard Studio option not present in some 8.2.0 upgrades)

 

# location of splunkd; don't include http[s]:// in this anymore.
mgmtHostPort = 127.0.0.1:8089

 

So, I reviewed $SPLUNK_HOME/etc/system/default/web.conf.  This line is still here!  Clearly this didn't prevent export in 8.5 and earlier but Splunk 9 doesn't like it any more.  I added an override in local/web.conf so

 

[settings]
httpport = 443
mgmtHostPort = <my server name>:8089
enableSplunkWebSSL = true

privKeyPath = /var/opt/<some file path>.key
serverCert = /var/opt/<some file path>.crt

 

This "fixed" the problem.

I'll consider this "solved" for now.  But I still have an unanswered question: My installation is a standalone server, NOT universal forwarder.  According to web.conf.spec,

mgmtHostPort = <string>
* The host port of the splunkd process.
* The IP address and host port where Splunk Web looks for the splunkd process.
* The port listens on all avalible host IP addresses (0.0.0.0)
* Don't include "http[s]://" when specifying this setting. Only
  include the IP address and port.
* Default (on universal forwarders): localhost:8089
* Default (on all other Splunk platform instance types): 0.0.0.0:8089 

In other words, installer should have set 0.0.0.0:8089 in etc/system/default/web.conf .  Why is it 127.0.0.1:8089?

Tags (1)
0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...