Deployment Architecture

Trying to install an SSL certificate on a search head, why is it getting stuck at "Waiting for web server at https://127.0.0.1:8000 to be available."?

j_kennedy
New Member

I'm trying to install an SSL certificate onto a search head and something is wrong. It'll start up with enableSplunkWebSSL set to "no" of course.

[more /opt/splunk/etc/system/local/web.conf]

    [settings]
    #enableSplunkWebSSL = no
    enableSplunkWebSSL = yes
    privKeyPath = /opt/splunk/etc/auth/splunkweb/splunk-search1_csuchico_edu.key
    caCertPath = /opt/splunk/etc/auth/splunkweb/splunk-search1_csuchico_edu.pem

It hands "Waiting for web server" with a single dot (vs "... available.. Done"):

    [bin/splunk start]
    Splunk> Needle. Haystack. Found.

    Checking prerequisites...
            Checking http port [8000]: open
            Checking mgmt port [8089]: open
            Checking appserver port [127.0.0.1:8065]: open
            Checking kvstore port [8191]: open
            Checking configuration...  Done.
            Checking critical directories...        Done
            Checking indexes...
                    Validated: _audit _internal _introspection _thefishbucket history main summary
            Done


    Bypassing local license checks since this instance is configured with a remote license master.

            Checking filesystem compatibility...  Done
            Checking conf files for problems...
            Done
            Checking default conf files for edits...
            Validating installed files against hashes from '/opt/splunk/splunk-6.3.3-f44afce176d0-linux-2.6-x86_64-manifest'
            All installed files intact.
            Done
    All preliminary checks passed.

    Starting splunk server daemon (splunkd)...
    Done
     [  OK  ]

    Waiting for web server at https://127.0.0.1:8000 to be available.

The new certificate is new/different because it has two intermediate certificates, but I strung it together like the documentation suggests, from cert to root, using the X509v3 Authority Key Identifier and X509v3 Subject Key Identifier to chain them together up to the self-signed AddTrust root CA:

[for FN in p4 p3 p2 p1; do openssl x509 -text < $FN | grep -i -e issuer: -e subject: -e keyid: -e 1E:05:A3:77:8F:6C:96:E2:5B:87:4B:A6:B4:86:AC:71:00:0C:E7:38 -e 53:79:BF:5A:AA:2B:4A:CF:54:80:E1:D8:9B:C0:9D:F2:B2:03:66:CB -e AD:BD:98:7A:34:B4:26:F7:FA:C4:26:54:EF:03:BD:E0:24:CB:54:1A; echo; done]

    Issuer: C=US, ST=MI, L=Ann Arbor, O=Internet2, OU=InCommon, CN=InCommon RSA Server CA
    Subject: C=US/postalCode=95929, ST=CA, L=Chico/street=400 W. 1st Street, O=California State University, Chico, OU=ISEC, CN=splunk-search1.csuchico.edu
            keyid:1E:05:A3:77:8F:6C:96:E2:5B:87:4B:A6:B4:86:AC:71:00:0C:E7:38

    Issuer: C=US, ST=New Jersey, L=Jersey City, O=The USERTRUST Network, CN=USERTrust RSA Certification Authority
    Subject: C=US, ST=MI, L=Ann Arbor, O=Internet2, OU=InCommon, CN=InCommon RSA Server CA
            keyid:53:79:BF:5A:AA:2B:4A:CF:54:80:E1:D8:9B:C0:9D:F2:B2:03:66:CB
            1E:05:A3:77:8F:6C:96:E2:5B:87:4B:A6:B4:86:AC:71:00:0C:E7:38

    Issuer: C=SE, O=AddTrust AB, OU=AddTrust External TTP Network, CN=AddTrust External CA Root
    Subject: C=US, ST=New Jersey, L=Jersey City, O=The USERTRUST Network, CN=USERTrust RSA Certification Authority
            keyid:AD:BD:98:7A:34:B4:26:F7:FA:C4:26:54:EF:03:BD:E0:24:CB:54:1A
            53:79:BF:5A:AA:2B:4A:CF:54:80:E1:D8:9B:C0:9D:F2:B2:03:66:CB

    Issuer: C=SE, O=AddTrust AB, OU=AddTrust External TTP Network, CN=AddTrust External CA Root
    Subject: C=SE, O=AddTrust AB, OU=AddTrust External TTP Network, CN=AddTrust External CA Root
            AD:BD:98:7A:34:B4:26:F7:FA:C4:26:54:EF:03:BD:E0:24:CB:54:1A
            keyid:AD:BD:98:7A:34:B4:26:F7:FA:C4:26:54:EF:03:BD:E0:24:CB:54:1A

I built my PEM like this, and they key has no password:

    cat p4 p3 p2 p1 > splunk-search1_csuchico_edu.pem

I'm not seeing any errors in the web_service.log.

Using openssl s_client -showcerts -connect localhost:8000 hands without showing any certs, much like the splunk-start seems to be doing.

Any suggestions on how I might debug this farther to try and figure out what needs to be fixed?

0 Karma

Tetonka
Engager

I have not had much luck with the web_service.log for troubleshooting. Can you restart splunk, then run a tail -300 on the /opt/splunk/var/log/splunk/splunkd.log and post the results here?

There should be an entry similar to this:

-0600 ERROR SSLCommon - Can't read certificate file /opt/splunk/etc/auth/MYSPLUNKCERTS & Key/splunk-Hostname-Private.key 
errno=33558530 error:02001002:system library:fopen:No such file or directory
-0600 ERROR HTTPServer - SSL context could not be created - error in cert or password is wrong
-0600 ERROR HTTPServer - SSL will not be enabled 
That may provide better suggest to debug
0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...