Trying to install an SSL certificate on a search head, why is it getting stuck at "Waiting for web server at https://127.0.0.1:8000 to be available."?

j_kennedy — Wed, 17 Feb 2016 23:29:47 GMT

I'm trying to install an SSL certificate onto a search head and something is wrong. It'll start up with enableSplunkWebSSL set to "no" of course.

[more /opt/splunk/etc/system/local/web.conf]

    [settings]
    #enableSplunkWebSSL = no
    enableSplunkWebSSL = yes
    privKeyPath = /opt/splunk/etc/auth/splunkweb/splunk-search1_csuchico_edu.key
    caCertPath = /opt/splunk/etc/auth/splunkweb/splunk-search1_csuchico_edu.pem

It hands "Waiting for web server" with a single dot (vs "... available.. Done"):

    [bin/splunk start]
    Splunk> Needle. Haystack. Found.

    Checking prerequisites...
            Checking http port [8000]: open
            Checking mgmt port [8089]: open
            Checking appserver port [127.0.0.1:8065]: open
            Checking kvstore port [8191]: open
            Checking configuration...  Done.
            Checking critical directories...        Done
            Checking indexes...
                    Validated: _audit _internal _introspection _thefishbucket history main summary
            Done


    Bypassing local license checks since this instance is configured with a remote license master.

            Checking filesystem compatibility...  Done
            Checking conf files for problems...
            Done
            Checking default conf files for edits...
            Validating installed files against hashes from '/opt/splunk/splunk-6.3.3-f44afce176d0-linux-2.6-x86_64-manifest'
            All installed files intact.
            Done
    All preliminary checks passed.

    Starting splunk server daemon (splunkd)...
    Done
     [  OK  ]

    Waiting for web server at https://127.0.0.1:8000 to be available.

The new certificate is new/different because it has two intermediate certificates, but I strung it together like the documentation suggests, from cert to root, using the X509v3 Authority Key Identifier and X509v3 Subject Key Identifier to chain them together up to the self-signed AddTrust root CA:

[for FN in p4 p3 p2 p1; do openssl x509 -text < $FN | grep -i -e issuer: -e subject: -e keyid: -e 1E:05:A3:77:8F:6C:96:E2:5B:87:4B:A6:B4:86:AC:71:00:0C:E7:38 -e 53:79:BF:5A:AA:2B:4A:CF:54:80:E1:D8:9B:C0:9D:F2:B2:03:66:CB -e AD:BD:98:7A:34:B4:26:F7:FA:C4:26:54:EF:03:BD:E0:24:CB:54:1A; echo; done]

    Issuer: C=US, ST=MI, L=Ann Arbor, O=Internet2, OU=InCommon, CN=InCommon RSA Server CA
    Subject: C=US/postalCode=95929, ST=CA, L=Chico/street=400 W. 1st Street, O=California State University, Chico, OU=ISEC, CN=splunk-search1.csuchico.edu
            keyid:1E:05:A3:77:8F:6C:96:E2:5B:87:4B:A6:B4:86:AC:71:00:0C:E7:38

    Issuer: C=US, ST=New Jersey, L=Jersey City, O=The USERTRUST Network, CN=USERTrust RSA Certification Authority
    Subject: C=US, ST=MI, L=Ann Arbor, O=Internet2, OU=InCommon, CN=InCommon RSA Server CA
            keyid:53:79:BF:5A:AA:2B:4A:CF:54:80:E1:D8:9B:C0:9D:F2:B2:03:66:CB
            1E:05:A3:77:8F:6C:96:E2:5B:87:4B:A6:B4:86:AC:71:00:0C:E7:38

    Issuer: C=SE, O=AddTrust AB, OU=AddTrust External TTP Network, CN=AddTrust External CA Root
    Subject: C=US, ST=New Jersey, L=Jersey City, O=The USERTRUST Network, CN=USERTrust RSA Certification Authority
            keyid:AD:BD:98:7A:34:B4:26:F7:FA:C4:26:54:EF:03:BD:E0:24:CB:54:1A
            53:79:BF:5A:AA:2B:4A:CF:54:80:E1:D8:9B:C0:9D:F2:B2:03:66:CB

    Issuer: C=SE, O=AddTrust AB, OU=AddTrust External TTP Network, CN=AddTrust External CA Root
    Subject: C=SE, O=AddTrust AB, OU=AddTrust External TTP Network, CN=AddTrust External CA Root
            AD:BD:98:7A:34:B4:26:F7:FA:C4:26:54:EF:03:BD:E0:24:CB:54:1A
            keyid:AD:BD:98:7A:34:B4:26:F7:FA:C4:26:54:EF:03:BD:E0:24:CB:54:1A

I built my PEM like this, and they key has no password:

    cat p4 p3 p2 p1 > splunk-search1_csuchico_edu.pem

I'm not seeing any errors in the web_service.log.

Using openssl s_client -showcerts -connect localhost:8000 hands without showing any certs, much like the splunk-start seems to be doing.

Any suggestions on how I might debug this farther to try and figure out what needs to be fixed?

Re: Trying to install an SSL certificate on a search head, why is it getting stuck at "Waiting for web server at https://127.0.0.1:8000 to be available."?

Tetonka — Thu, 03 Mar 2016 20:37:24 GMT

I have not had much luck with the web_service.log for troubleshooting. Can you restart splunk, then run a tail -300 on the /opt/splunk/var/log/splunk/splunkd.log and post the results here?

There should be an entry similar to this:

-0600 ERROR SSLCommon - Can't read certificate file /opt/splunk/etc/auth/MYSPLUNKCERTS & Key/splunk-Hostname-Private.key 
errno=33558530 error:02001002:system library:fopen:No such file or directory
-0600 ERROR HTTPServer - SSL context could not be created - error in cert or password is wrong
-0600 ERROR HTTPServer - SSL will not be enabled 
That may provide better suggest to debug

topic Re: Trying to install an SSL certificate on a search head, why is it getting stuck at "Waiting for web server at https://127.0.0.1:8000 to be available."? in Deployment Architecture

Trying to install an SSL certificate on a search head, why is it getting stuck at "Waiting for web server at https://127.0.0.1:8000 to be available."?

Re: Trying to install an SSL certificate on a search head, why is it getting stuck at "Waiting for web server at https://127.0.0.1:8000 to be available."?