topic Re: two SSL Certificates on a single indexer / forwarder in Security

two SSL Certificates on a single indexer / forwarder

inventsekar — Tue, 29 Sep 2020 14:34:03 GMT

Hi All,
1. can we deploy two SSL certificates (self signed) on a single indexer(either same port/different port)
2. for Certificates renewal, I have been thinking this process - can one UF be deployed with two SSL Certificates?
please suggest regarding this below process.. any improvements/suggestions please.

===================================================================
Normal situation (SSL deployed)

Indexer1(IP - 10.11.12.13) ------- Indexer2 (IP-10.11.12.14)
Inputs.conf (SSL-Cert-1) -------- inputs.conf (SSL- Cert 1)

UF1 UF2
outputs.conf file
[tcpout:splunkssl]
server = 10.11.12.13:9997, 10.11.12.14:9997
compressed = true
sslVerifyServerCert = false
sslRootCAPath = $SPLUNK_HOME/etc/auth/cacert.pem
sslCertPath = $SPLUNK_HOME/etc/auth/server.pem
sslPassword = password

===================================================================
During Certificate renewal -

Indexer1(IP - 10.11.12.13) ------------------ Indexer2 (IP-10.11.12.14)
Inputs.conf (SSL-Cert-1) --------------------- inputs.conf (SSL- Cert 2)

UF1 UF2
outputs.conf file with two SSL info updated –

[tcpout-server://10.11.12.13:9997]
sslRootCAPath = $SPLUNK_HOME/etc/certs/myCACertificate1.pem
sslCertPath = $SPLUNK_HOME/etc/certs/myServerCertificate1.pem
sslPassword = server_privkey_password
sslVerifyServerCert = true
sslCommonNameToCheck = phobos

[tcpout-server://10.11.12.14:9997]
sslRootCAPath = $SPLUNK_HOME/etc/certs/myCACertificate2.pem
sslCertPath = $SPLUNK_HOME/etc/certs/myServerCertificate2.pem
sslPassword = server_privkey_password
sslVerifyServerCert = true
sslCommonNameToCheck = deimos

Ref - https://wiki.splunk.com/Community:Splunk2Splunk_SSL_SelfSignedCert_NewRootCA

===================================================================

After Certificate renewal -

Indexer1(IP - 10.11.12.13) ------------------ Indexer2 (IP-10.11.12.14)
Inputs.conf (SSL-Cert-2) --------------------- inputs.conf (SSL- Cert 2)

UF1 UF2
outputs.conf file with two SSL info updated –

[tcpout:splunkssl]
server = 10.11.12.13:9997, 10.11.12.14:9997
compressed = true
sslVerifyServerCert = false
sslRootCAPath = $SPLUNK_HOME/etc/certs/myCACertificate2.pem
sslCertPath = $SPLUNK_HOME/etc/certs/myServerCertificate2.pem
sslPassword = password

===================================================================

Re: two SSL Certificates on a single indexer / forwarder

jkat54 — Tue, 29 Sep 2020 14:34:23 GMT

Yes you can deploy more than 1 ssl cert on a single indexer (on different ports)

Inputs.conf

[splunktcp-ssl:9997]
serverCert = /path/to/port/9997/cert
sslPassword = /path/to/port/9997/cert/key
sslCommonNameToCheck = forwarderCertCommonNames
compressed = true

[splunktcp-ssl:9998]
serverCert = /path/to/port/9998/cert
sslPassword = /path/to/port/9998/cert/key
sslCommonNameToCheck = forwarderCertCommonNames
compressed = true
Yes you can configure a forwarder to use two certs. Please note splunk will switch between the two often (aka auto load balance, but if one is down it will stick to whatever is up).

Outputs.conf (on forwarders):

[tcpout]
defaultGroup = secureIndexers

[tcpout:secureIndexers]
servers=10.11.12.13:9997, 10.11.12.14:9997

[tcpout-server://10.11.12.13:9997]
sslRootCAPath = $SPLUNK_HOME/etc/certs/myCACertificate1.pem
sslCertPath = $SPLUNK_HOME/etc/certs/myServerCertificate1.pem
sslPassword = server_privkey_password
sslVerifyServerCert = true
sslCommonNameToCheck = phobos
compressed = true

[tcpout-server://10.11.12.14:9997]
sslRootCAPath = $SPLUNK_HOME/etc/certs/myCACertificate2.pem
sslCertPath = $SPLUNK_HOME/etc/certs/myServerCertificate2.pem
sslPassword = server_privkey_password
sslVerifyServerCert = true
sslCommonNameToCheck = deimos
compressed = true

Inputs.conf (on Indexer named phobos)

[splunktcp-ssl:9997]
serverCert = /path/to/phobos/cert
sslPassword = /path/to/phobos/cert/key
sslCommonNameToCheck = forwarderCertCommonNames
compressed = true

Inputs.conf (on Indexer named deimos)

[splunktcp-ssl:9997]
serverCert = /path/to/deimos/cert
sslPassword = /path/to/deimos/cert/key
sslCommonNameToCheck = forwarderCertCommonNames
compressed = true
How to test your certs are good:

Verifying SSL Certificates on Remote / Local Machines
This will test the entire certificate chain to see if it’s verifiable or not.

openssl s_client -connect {server}:{port}

Port 8000, 8060, 8089, 9998, etc.

results should end with something like this:
Verify return code: 0 (ok)
---
if there are any errors above, the ssl cert will not validate/verify and the certificate will not be trusted

3.b. Verifying Completed Certificate (CER, CRT, or PEM) Matches Private or Public Keys

openssl x509  -noout -modulus -in {servername}.{yourdomain}.{yourtld}.{cer/crt/pem}| openssl md5
(stdin)= {MATCHING HASH}
openssl rsa -noout -modulus -in {servername}.{yourdomain}.{yourtld}.private.key | openssl md5
Enter pass phrase for {servername}.{yourdomain}.{yourtld}.private.key:
(stdin)= {MATCHING HASH}
openssl rsa -noout -modulus -in {servername}.{yourdomain}.{yourtld}.public.key  | openssl md5
(stdin)= {MATCHING HASH}

So perhaps the most robust design would be 4 certs per forwarder, 2 certs per indexer:

indexer1:9997 - Expires in 1 year
indexer1:9998 - Expires in 2 years
indexer2:9997 - Expires in 1 year
indexer2:9998 - Expires in 2 years

So that when one set expires, there are still two secure paths to indexers to fall back on.

Re: two SSL Certificates on a single indexer / forwarder

inventsekar — Sat, 24 Jun 2017 02:07:42 GMT

Hi Jkat54, much appreciated your reply.. let me test this.

Re: two SSL Certificates on a single indexer / forwarder

koshyk — Sun, 25 Jun 2017 09:59:52 GMT

Another way we tried is to install another instance of Splunk in the same server during transition phase on another port. (eg . splunk-instance-02). This helped us to completely isolate the complications and during the transition we moved all the new certificate connections to the new instance and once everything has been updated, we shifted back to the original splunk instance

high level steps
- deployment-server-main (/opt/splunk => mgmt:8089) + deployer-server-temporary (/opt/splunk2 => mgmt:9089) on same box
- Deploy same etc/deployment-apps in both the instances
- Push deployment-client app to the new certificate clients with mgmt:9089
- Get connectivity et.al working correctly
- After all 100% servers migrated, just push a the "deployment-client" app with mgmt:8089 address in it
- viola everything works. Now delete /opt/splunk2 instance.

Re: two SSL Certificates on a single indexer / forwarder

inventsekar — Wed, 28 Jun 2017 04:30:39 GMT

will be completing this in a day or two.. will update soon.

Re: two SSL Certificates on a single indexer / forwarder

inventsekar — Wed, 28 Jun 2017 04:32:18 GMT

installing another splunk instance --- that looks like a good option, but can projects afford a separate server - is a big concern. well, anyhow, thanks for your idea.

Re: two SSL Certificates on a single indexer / forwarder

jkat54 — Wed, 28 Jun 2017 20:09:58 GMT

I hope it all went well. SSL deployments are always "fun"!

Re: two SSL Certificates on a single indexer / forwarder

koshyk — Thu, 29 Jun 2017 14:25:28 GMT

hi inventsekar, I mean within the SAME server as another splunk instance. (/opt/splunk2). So you can keep the original instance isolated

Re: two SSL Certificates on a single indexer / forwarder

inventsekar — Thu, 06 Jul 2017 11:31:16 GMT

i awarded the 50 points, and then i accepted as answer.
they should have given instruction that - "when we accept as answer, the award points automatically will be given".
sad part, i thought if i unaccept, the 50 points will be back to me, but unaccept works, but not the 50 points 😉

now, only if @jkat54 award me back 50 points, i will get my 50. (i will accept this as an answer and he will get the 50)

Re: two SSL Certificates on a single indexer / forwarder

jkat54 — Thu, 06 Jul 2017 12:06:38 GMT

Yeah no problem man. Done.

Re: two SSL Certificates on a single indexer / forwarder

arrangineni — Fri, 19 Jul 2019 02:06:18 GMT

Can we setup two different SSL cert's on same Indexer using same port and configure few Forwarders to use the new certs and other forwarders to use the existing SSL certs as per below plan?

Current:
SSL cert1 > 5 Indexers and all the Forwarders

Test:
SSL cert1 > 5 Indexers and 100 forwarders
SSL cert2 > 5 Indexers and 50 forwarders

After:
SSL cert2 > 5 Indexers and 150 forwarders

Please suggest if we have any process for this approach

Re: two SSL Certificates on a single indexer / forwarder

jkat54 — Fri, 19 Jul 2019 04:20:51 GMT

Please post a new question. You can point to this question in your new question if you like.

topic Re: two SSL Certificates on a single indexer / forwarder in Security

two SSL Certificates on a single indexer / forwarder

Re: two SSL Certificates on a single indexer / forwarder

Inputs.conf

Outputs.conf (on forwarders):

Inputs.conf (on Indexer named phobos)

Inputs.conf (on Indexer named deimos)

Re: two SSL Certificates on a single indexer / forwarder

Re: two SSL Certificates on a single indexer / forwarder

Re: two SSL Certificates on a single indexer / forwarder

Re: two SSL Certificates on a single indexer / forwarder

Re: two SSL Certificates on a single indexer / forwarder

Re: two SSL Certificates on a single indexer / forwarder

Re: two SSL Certificates on a single indexer / forwarder

Re: two SSL Certificates on a single indexer / forwarder

Re: two SSL Certificates on a single indexer / forwarder

Re: two SSL Certificates on a single indexer / forwarder