OpenSSL VERIFY_NONE, Splunk Heavy Forwarder, db_connect, Chef, Ruby, vulnerability remediation

[tilburn]

Hi friends:

Looking for some assistance from savvy folks with heavy forwarders / db_connect  / ssl experience.

Environment:

[root@ip-10-105-90-142 bin]# ./splunk -version
Splunk 8.0.3 (build a6754d8441bf)
Splunk Heavy Forwarder
apps: db_connect
certs: all are default splunk certs which come from 8.0.3 app archive which gets freshly deployed each month.


Chef code deployment block which is generating vulnerability (which I inherited):

ruby_block 'Wait for Splunk DBConnect to respond' do
  block do
    uri = URI('https://localhost:8089/servicesNS/nobody/splunk_app_db_connect/db_connect/')
    req = Net::HTTP::Get.new(uri.path)
    req.basic_auth 'admin', 'changeme'
    return_code = '0'
    until return_code == '200'
      Net::HTTP.start(
        uri.host, uri.port,
        use_ssl: uri.scheme == 'https',
        verify_mode: OpenSSL::SSL::VERIFY_NONE) do |https|
        return_code = https.request(req).code
      end
      sleep(3)
    end
  end
  action :run
  not_if { File.exist? '/opt/splunk/etc/apps/splunk_app_db_connect/local/identities.conf' }
end


Goal:
Clearly the goal is to resolve the bad coding of VERIFY_NONE on the verify_mode line


Effort:

When I deploy the code as-is, it works fine but is vulnerable as noted.  

When I update VERIFY_NONE to VERIFY_PEER, the code will technically deploy ok - but this value throws an error which causes the Jenkins pipeline job to fail, so simply accepting the error and moving forward isn't an option. (Even when I use "ignore_failure true" in the chef resource block.), not that you would want to do that anyway.

The error it throws (fromo chef output log) is:
  * ruby_block[Wait for Splunk DBConnect to respond] action run[2021-06-08T18:04:42+00:00] ERROR: ruby_block[Wait for Splunk DBConnect to respond] (efg::splunk_heavy line 277) had an error: OpenSSL::SSL::SSLError: SSL_connect returned=1 errno=0 state=error: certificate verify failed (self signed certificate in certificate chain); ignore_failure is set, continuing


It looks like a local self signed cert issue, which should be easily resolvable by applying it in the config. However, I've been at this a few days now, and am still not able to figure it out.

This code that I inherited looks like it's trying to VERIFY_ at the db_connect app config.  Does that seem correct/logical?  it's purpose is to wait for db_connect to be reachable before it creates the connection to the DB.


The cert that it is using looks ok:

[root@ip-10-105-90-142 etc]# grep server.pem /opt/splunk/etc/system/default/server.conf
serverCert = $SPLUNK_HOME/etc/auth/server.pem

[root@ip-10-105-90-142 auth]# openssl x509 -in /opt/splunk/etc/auth/server.pem -noout -text | grep "Not After"
            Not After : Jun  4 07:30:32 2024 GMT


However, checking the local server cert, I get the Verify Return Code 19 error:

[root@ip-10-105-90-142 ~]# openssl s_client -connect 10.105.90.142:8089 | head 
depth=1 C = US, ST = CA, L = San Francisco, O = Splunk, CN = SplunkCommonCA, emailAddress = support@splunk.com
verify error:num=19:self signed certificate in certificate chain
CONNECTED(00000003)
---
Certificate chain
 0 s:/CN=SplunkServerDefaultCert/O=SplunkUser
   i:/C=US/ST=CA/L=San Francisco/O=Splunk/CN=SplunkCommonCA/emailAddress=support@splunk.com
 1 s:/C=US/ST=CA/L=San Francisco/O=Splunk/CN=SplunkCommonCA/emailAddress=support@splunk.com
   i:/C=US/ST=CA/L=San Francisco/O=Splunk/CN=SplunkCommonCA/emailAddress=support@splunk.com
---
Server certificate
-----BEGIN CERTIFICATE-----

This error would explain why the chef code ruby_block fails.  This is my current focus that I'm not quite able to remediate.
 
Oddly, the output of that openssl command that yields that cert info does not reveal any expiry date.  I find that odd and am not sure why that's the case.



Questions:
1. Are this openssl check and the code block consistent with respect to what cert they are trying to retrieve? i.e. is there a different local cert for app_db_connect config?
2. Am I correct that the local server openssl check (above) is the same ssl check that the code is performing and waiting for? (verify_mode line of code) 



Appreciate any insight or help anyone can lend.

Many thanks,
Bill

What am I missing to validate with VERIFY_PEER openssl method?