Security

two SSL Certificates on a single indexer / forwarder

Champion

Hi All,
1. can we deploy two SSL certificates (self signed) on a single indexer(either same port/different port)
2. for Certificates renewal, I have been thinking this process - can one UF be deployed with two SSL Certificates?
please suggest regarding this below process.. any improvements/suggestions please.

===================================================================
Normal situation (SSL deployed)

Indexer1(IP - 10.11.12.13) ------- Indexer2 (IP-10.11.12.14)
Inputs.conf (SSL-Cert-1) -------- inputs.conf (SSL- Cert 1)

UF1 UF2
outputs.conf file
[tcpout:splunkssl]
server = 10.11.12.13:9997, 10.11.12.14:9997
compressed = true
sslVerifyServerCert = false
sslRootCAPath = $SPLUNKHOME/etc/auth/cacert.pem
sslCertPath = $SPLUNK
HOME/etc/auth/server.pem
sslPassword = password

===================================================================
During Certificate renewal -

Indexer1(IP - 10.11.12.13) ------------------ Indexer2 (IP-10.11.12.14)
Inputs.conf (SSL-Cert-1) --------------------- inputs.conf (SSL- Cert 2)

UF1 UF2
outputs.conf file with two SSL info updated –

[tcpout-server://10.11.12.13:9997]
sslRootCAPath = $SPLUNKHOME/etc/certs/myCACertificate1.pem
sslCertPath = $SPLUNK
HOME/etc/certs/myServerCertificate1.pem
sslPassword = serverprivkeypassword
sslVerifyServerCert = true
sslCommonNameToCheck = phobos

[tcpout-server://10.11.12.14:9997]
sslRootCAPath = $SPLUNKHOME/etc/certs/myCACertificate2.pem
sslCertPath = $SPLUNK
HOME/etc/certs/myServerCertificate2.pem
sslPassword = serverprivkeypassword
sslVerifyServerCert = true
sslCommonNameToCheck = deimos

Ref - https://wiki.splunk.com/Community:Splunk2Splunk_SSL_SelfSignedCert_NewRootCA

===================================================================

After Certificate renewal -

Indexer1(IP - 10.11.12.13) ------------------ Indexer2 (IP-10.11.12.14)
Inputs.conf (SSL-Cert-2) --------------------- inputs.conf (SSL- Cert 2)

UF1 UF2
outputs.conf file with two SSL info updated –

[tcpout:splunkssl]
server = 10.11.12.13:9997, 10.11.12.14:9997
compressed = true
sslVerifyServerCert = false
sslRootCAPath = $SPLUNKHOME/etc/certs/myCACertificate2.pem
sslCertPath = $SPLUNK
HOME/etc/certs/myServerCertificate2.pem
sslPassword = password

===================================================================

1 Solution

SplunkTrust
SplunkTrust
  1. Yes you can deploy more than 1 ssl cert on a single indexer (on different ports)

    Inputs.conf

    [splunktcp-ssl:9997]
    serverCert = /path/to/port/9997/cert
    sslPassword = /path/to/port/9997/cert/key
    sslCommonNameToCheck = forwarderCertCommonNames
    compressed = true

    [splunktcp-ssl:9998]
    serverCert = /path/to/port/9998/cert
    sslPassword = /path/to/port/9998/cert/key
    sslCommonNameToCheck = forwarderCertCommonNames
    compressed = true

  2. Yes you can configure a forwarder to use two certs. Please note splunk will switch between the two often (aka auto load balance, but if one is down it will stick to whatever is up).

    Outputs.conf (on forwarders):

    [tcpout]
    defaultGroup = secureIndexers

    [tcpout:secureIndexers]
    servers=10.11.12.13:9997, 10.11.12.14:9997

    [tcpout-server://10.11.12.13:9997]
    sslRootCAPath = $SPLUNKHOME/etc/certs/myCACertificate1.pem
    sslCertPath = $SPLUNK
    HOME/etc/certs/myServerCertificate1.pem
    sslPassword = serverprivkeypassword
    sslVerifyServerCert = true
    sslCommonNameToCheck = phobos
    compressed = true

    [tcpout-server://10.11.12.14:9997]
    sslRootCAPath = $SPLUNKHOME/etc/certs/myCACertificate2.pem
    sslCertPath = $SPLUNK
    HOME/etc/certs/myServerCertificate2.pem
    sslPassword = serverprivkeypassword
    sslVerifyServerCert = true
    sslCommonNameToCheck = deimos
    compressed = true

    Inputs.conf (on Indexer named phobos)

    [splunktcp-ssl:9997]
    serverCert = /path/to/phobos/cert
    sslPassword = /path/to/phobos/cert/key
    sslCommonNameToCheck = forwarderCertCommonNames
    compressed = true

    Inputs.conf (on Indexer named deimos)

    [splunktcp-ssl:9997]
    serverCert = /path/to/deimos/cert
    sslPassword = /path/to/deimos/cert/key
    sslCommonNameToCheck = forwarderCertCommonNames
    compressed = true

  3. How to test your certs are good:

    Verifying SSL Certificates on Remote / Local Machines
    This will test the entire certificate chain to see if it’s verifiable or not.

    openssl s_client -connect {server}:{port}

    Port 8000, 8060, 8089, 9998, etc.

    results should end with something like this:
    Verify return code: 0 (ok)
    ---
    if there are any errors above, the ssl cert will not validate/verify and the certificate will not be trusted

3.b. Verifying Completed Certificate (CER, CRT, or PEM) Matches Private or Public Keys

openssl x509  -noout -modulus -in {servername}.{yourdomain}.{yourtld}.{cer/crt/pem}| openssl md5
(stdin)= {MATCHING HASH}
openssl rsa -noout -modulus -in {servername}.{yourdomain}.{yourtld}.private.key | openssl md5
Enter pass phrase for {servername}.{yourdomain}.{yourtld}.private.key:
(stdin)= {MATCHING HASH}
openssl rsa -noout -modulus -in {servername}.{yourdomain}.{yourtld}.public.key  | openssl md5
(stdin)= {MATCHING HASH}

So perhaps the most robust design would be 4 certs per forwarder, 2 certs per indexer:

indexer1:9997 - Expires in 1 year
indexer1:9998 - Expires in 2 years
indexer2:9997 - Expires in 1 year
indexer2:9998 - Expires in 2 years

So that when one set expires, there are still two secure paths to indexers to fall back on.

View solution in original post

Path Finder

Can we setup two different SSL cert's on same Indexer using same port and configure few Forwarders to use the new certs and other forwarders to use the existing SSL certs as per below plan?

Current:
SSL cert1 > 5 Indexers and all the Forwarders

Test:
SSL cert1 > 5 Indexers and 100 forwarders
SSL cert2 > 5 Indexers and 50 forwarders

After:
SSL cert2 > 5 Indexers and 150 forwarders

Please suggest if we have any process for this approach

0 Karma

SplunkTrust
SplunkTrust

Please post a new question. You can point to this question in your new question if you like.

0 Karma

Super Champion

Another way we tried is to install another instance of Splunk in the same server during transition phase on another port. (eg . splunk-instance-02). This helped us to completely isolate the complications and during the transition we moved all the new certificate connections to the new instance and once everything has been updated, we shifted back to the original splunk instance

high level steps
- deployment-server-main (/opt/splunk => mgmt:8089) + deployer-server-temporary (/opt/splunk2 => mgmt:9089) on same box
- Deploy same etc/deployment-apps in both the instances
- Push deployment-client app to the new certificate clients with mgmt:9089
- Get connectivity et.al working correctly
- After all 100% servers migrated, just push a the "deployment-client" app with mgmt:8089 address in it
- viola everything works. Now delete /opt/splunk2 instance.

Champion

installing another splunk instance --- that looks like a good option, but can projects afford a separate server - is a big concern. well, anyhow, thanks for your idea.

0 Karma

Super Champion

hi inventsekar, I mean within the SAME server as another splunk instance. (/opt/splunk2). So you can keep the original instance isolated

0 Karma

SplunkTrust
SplunkTrust
  1. Yes you can deploy more than 1 ssl cert on a single indexer (on different ports)

    Inputs.conf

    [splunktcp-ssl:9997]
    serverCert = /path/to/port/9997/cert
    sslPassword = /path/to/port/9997/cert/key
    sslCommonNameToCheck = forwarderCertCommonNames
    compressed = true

    [splunktcp-ssl:9998]
    serverCert = /path/to/port/9998/cert
    sslPassword = /path/to/port/9998/cert/key
    sslCommonNameToCheck = forwarderCertCommonNames
    compressed = true

  2. Yes you can configure a forwarder to use two certs. Please note splunk will switch between the two often (aka auto load balance, but if one is down it will stick to whatever is up).

    Outputs.conf (on forwarders):

    [tcpout]
    defaultGroup = secureIndexers

    [tcpout:secureIndexers]
    servers=10.11.12.13:9997, 10.11.12.14:9997

    [tcpout-server://10.11.12.13:9997]
    sslRootCAPath = $SPLUNKHOME/etc/certs/myCACertificate1.pem
    sslCertPath = $SPLUNK
    HOME/etc/certs/myServerCertificate1.pem
    sslPassword = serverprivkeypassword
    sslVerifyServerCert = true
    sslCommonNameToCheck = phobos
    compressed = true

    [tcpout-server://10.11.12.14:9997]
    sslRootCAPath = $SPLUNKHOME/etc/certs/myCACertificate2.pem
    sslCertPath = $SPLUNK
    HOME/etc/certs/myServerCertificate2.pem
    sslPassword = serverprivkeypassword
    sslVerifyServerCert = true
    sslCommonNameToCheck = deimos
    compressed = true

    Inputs.conf (on Indexer named phobos)

    [splunktcp-ssl:9997]
    serverCert = /path/to/phobos/cert
    sslPassword = /path/to/phobos/cert/key
    sslCommonNameToCheck = forwarderCertCommonNames
    compressed = true

    Inputs.conf (on Indexer named deimos)

    [splunktcp-ssl:9997]
    serverCert = /path/to/deimos/cert
    sslPassword = /path/to/deimos/cert/key
    sslCommonNameToCheck = forwarderCertCommonNames
    compressed = true

  3. How to test your certs are good:

    Verifying SSL Certificates on Remote / Local Machines
    This will test the entire certificate chain to see if it’s verifiable or not.

    openssl s_client -connect {server}:{port}

    Port 8000, 8060, 8089, 9998, etc.

    results should end with something like this:
    Verify return code: 0 (ok)
    ---
    if there are any errors above, the ssl cert will not validate/verify and the certificate will not be trusted

3.b. Verifying Completed Certificate (CER, CRT, or PEM) Matches Private or Public Keys

openssl x509  -noout -modulus -in {servername}.{yourdomain}.{yourtld}.{cer/crt/pem}| openssl md5
(stdin)= {MATCHING HASH}
openssl rsa -noout -modulus -in {servername}.{yourdomain}.{yourtld}.private.key | openssl md5
Enter pass phrase for {servername}.{yourdomain}.{yourtld}.private.key:
(stdin)= {MATCHING HASH}
openssl rsa -noout -modulus -in {servername}.{yourdomain}.{yourtld}.public.key  | openssl md5
(stdin)= {MATCHING HASH}

So perhaps the most robust design would be 4 certs per forwarder, 2 certs per indexer:

indexer1:9997 - Expires in 1 year
indexer1:9998 - Expires in 2 years
indexer2:9997 - Expires in 1 year
indexer2:9998 - Expires in 2 years

So that when one set expires, there are still two secure paths to indexers to fall back on.

View solution in original post

Champion

i awarded the 50 points, and then i accepted as answer.
they should have given instruction that - "when we accept as answer, the award points automatically will be given".
sad part, i thought if i unaccept, the 50 points will be back to me, but unaccept works, but not the 50 points 😉

now, only if @jkat54 award me back 50 points, i will get my 50. (i will accept this as an answer and he will get the 50)

0 Karma

SplunkTrust
SplunkTrust

Yeah no problem man. Done.

0 Karma

Champion

Hi Jkat54, much appreciated your reply.. let me test this.

0 Karma

SplunkTrust
SplunkTrust

I hope it all went well. SSL deployments are always "fun"!

0 Karma

Champion

will be completing this in a day or two.. will update soon.

0 Karma