two SSL Certificates on a single indexer / forwarder

[inventsekar]

Hi All,
1. can we deploy two SSL certificates (self signed) on a single indexer(either same port/different port)
2. for Certificates renewal, I have been thinking this process - can one UF be deployed with two SSL Certificates?
please suggest regarding this below process.. any improvements/suggestions please.

===================================================================
Normal situation (SSL deployed)

Indexer1(IP - 10.11.12.13) ------- Indexer2 (IP-10.11.12.14)
Inputs.conf (SSL-Cert-1) -------- inputs.conf (SSL- Cert 1)

UF1 UF2
outputs.conf file
[tcpout:splunkssl]
server = 10.11.12.13:9997, 10.11.12.14:9997
compressed = true
sslVerifyServerCert = false
sslRootCAPath = $SPLUNK_HOME/etc/auth/cacert.pem
sslCertPath = $SPLUNK_HOME/etc/auth/server.pem
sslPassword = password

===================================================================
During Certificate renewal -

Indexer1(IP - 10.11.12.13) ------------------ Indexer2 (IP-10.11.12.14)
Inputs.conf (SSL-Cert-1) --------------------- inputs.conf (SSL- Cert 2)

UF1 UF2
outputs.conf file with two SSL info updated –

[tcpout-server://10.11.12.13:9997]
sslRootCAPath = $SPLUNK_HOME/etc/certs/myCACertificate1.pem
sslCertPath = $SPLUNK_HOME/etc/certs/myServerCertificate1.pem
sslPassword = server_privkey_password
sslVerifyServerCert = true
sslCommonNameToCheck = phobos

[tcpout-server://10.11.12.14:9997]
sslRootCAPath = $SPLUNK_HOME/etc/certs/myCACertificate2.pem
sslCertPath = $SPLUNK_HOME/etc/certs/myServerCertificate2.pem
sslPassword = server_privkey_password
sslVerifyServerCert = true
sslCommonNameToCheck = deimos

Ref - https://wiki.splunk.com/Community:Splunk2Splunk_SSL_SelfSignedCert_NewRootCA

===================================================================

After Certificate renewal -

Indexer1(IP - 10.11.12.13) ------------------ Indexer2 (IP-10.11.12.14)
Inputs.conf (SSL-Cert-2) --------------------- inputs.conf (SSL- Cert 2)

UF1 UF2
outputs.conf file with two SSL info updated –

[tcpout:splunkssl]
server = 10.11.12.13:9997, 10.11.12.14:9997
compressed = true
sslVerifyServerCert = false
sslRootCAPath = $SPLUNK_HOME/etc/certs/myCACertificate2.pem
sslCertPath = $SPLUNK_HOME/etc/certs/myServerCertificate2.pem
sslPassword = password

===================================================================

[arrangineni]

Can we setup two different SSL cert's on same Indexer using same port and configure few Forwarders to use the new certs and other forwarders to use the existing SSL certs as per below plan?

Current:
SSL cert1 > 5 Indexers and all the Forwarders

Test:
SSL cert1 > 5 Indexers and 100 forwarders
SSL cert2 > 5 Indexers and 50 forwarders

After:
SSL cert2 > 5 Indexers and 150 forwarders

Please suggest if we have any process for this approach

[jkat54]

Please post a new question. You can point to this question in your new question if you like.

[koshyk]

Another way we tried is to install another instance of Splunk in the same server during transition phase on another port. (eg . splunk-instance-02). This helped us to completely isolate the complications and during the transition we moved all the new certificate connections to the new instance and once everything has been updated, we shifted back to the original splunk instance

high level steps
- deployment-server-main (/opt/splunk => mgmt:8089) + deployer-server-temporary (/opt/splunk2 => mgmt:9089) on same box
- Deploy same etc/deployment-apps in both the instances
- Push deployment-client app to the new certificate clients with mgmt:9089
- Get connectivity et.al working correctly
- After all 100% servers migrated, just push a the "deployment-client" app with mgmt:8089 address in it
- viola everything works. Now delete /opt/splunk2 instance.

[inventsekar]

installing another splunk instance --- that looks like a good option, but can projects afford a separate server - is a big concern. well, anyhow, thanks for your idea.

[koshyk]

hi inventsekar, I mean within the SAME server as another splunk instance. (/opt/splunk2). So you can keep the original instance isolated

[jkat54 · Accepted solution]

1. Yes you can deploy more than 1 ssl cert on a single indexer (on different ports)

   Inputs.conf

   [splunktcp-ssl:9997]
   serverCert = /path/to/port/9997/cert
   sslPassword = /path/to/port/9997/cert/key
   sslCommonNameToCheck = forwarderCertCommonNames
   compressed = true

   [splunktcp-ssl:9998]
   serverCert = /path/to/port/9998/cert
   sslPassword = /path/to/port/9998/cert/key
   sslCommonNameToCheck = forwarderCertCommonNames
   compressed = true

2. Yes you can configure a forwarder to use two certs. Please note splunk will switch between the two often (aka auto load balance, but if one is down it will stick to whatever is up).

   Outputs.conf (on forwarders):

   [tcpout]
   defaultGroup = secureIndexers

   [tcpout:secureIndexers]
   servers=10.11.12.13:9997, 10.11.12.14:9997

   [tcpout-server://10.11.12.13:9997]
   sslRootCAPath = $SPLUNK_HOME/etc/certs/myCACertificate1.pem
   sslCertPath = $SPLUNK_HOME/etc/certs/myServerCertificate1.pem
   sslPassword = server_privkey_password
   sslVerifyServerCert = true
   sslCommonNameToCheck = phobos
   compressed = true

   [tcpout-server://10.11.12.14:9997]
   sslRootCAPath = $SPLUNK_HOME/etc/certs/myCACertificate2.pem
   sslCertPath = $SPLUNK_HOME/etc/certs/myServerCertificate2.pem
   sslPassword = server_privkey_password
   sslVerifyServerCert = true
   sslCommonNameToCheck = deimos
   compressed = true

   Inputs.conf (on Indexer named phobos)

   [splunktcp-ssl:9997]
   serverCert = /path/to/phobos/cert
   sslPassword = /path/to/phobos/cert/key
   sslCommonNameToCheck = forwarderCertCommonNames
   compressed = true

   Inputs.conf (on Indexer named deimos)

   [splunktcp-ssl:9997]
   serverCert = /path/to/deimos/cert
   sslPassword = /path/to/deimos/cert/key
   sslCommonNameToCheck = forwarderCertCommonNames
   compressed = true

3. How to test your certs are good:

   Verifying SSL Certificates on Remote / Local Machines
   This will test the entire certificate chain to see if it’s verifiable or not.

   openssl s_client -connect {server}:{port}

   Port 8000, 8060, 8089, 9998, etc.

   results should end with something like this:
   Verify return code: 0 (ok)
   ---
   if there are any errors above, the ssl cert will not validate/verify and the certificate will not be trusted

3.b. Verifying Completed Certificate (CER, CRT, or PEM) Matches Private or Public Keys

openssl x509  -noout -modulus -in {servername}.{yourdomain}.{yourtld}.{cer/crt/pem}| openssl md5
(stdin)= {MATCHING HASH}
openssl rsa -noout -modulus -in {servername}.{yourdomain}.{yourtld}.private.key | openssl md5
Enter pass phrase for {servername}.{yourdomain}.{yourtld}.private.key:
(stdin)= {MATCHING HASH}
openssl rsa -noout -modulus -in {servername}.{yourdomain}.{yourtld}.public.key  | openssl md5
(stdin)= {MATCHING HASH}

So perhaps the most robust design would be 4 certs per forwarder, 2 certs per indexer:

indexer1:9997 - Expires in 1 year
indexer1:9998 - Expires in 2 years
indexer2:9997 - Expires in 1 year
indexer2:9998 - Expires in 2 years

So that when one set expires, there are still two secure paths to indexers to fall back on.

[inventsekar]

i awarded the 50 points, and then i accepted as answer.
they should have given instruction that - "when we accept as answer, the award points automatically will be given".
sad part, i thought if i unaccept, the 50 points will be back to me, but unaccept works, but not the 50 points 😉

now, only if @jkat54 award me back 50 points, i will get my 50. (i will accept this as an answer and he will get the 50)

[jkat54]

Yeah no problem man. Done.

[inventsekar]

Hi Jkat54, much appreciated your reply.. let me test this.

[jkat54]

I hope it all went well. SSL deployments are always "fun"!

[inventsekar]

will be completing this in a day or two.. will update soon.