I have a customer that is evaluating Splunk in a cloud provider. They are trying to evaluate the performance of bare metal vs VM instances.
There are four hosts
1 bare metal Enterprise indexer
1 VM Enterprise indexer
2 VM forwarders configured to send one copy
I have created self-signed certs for all of the hosts --- web, forwarding, etc. The SSL config in server.conf is identical for both of them except for the name of the server certificate.
Today on the bare metal instance, the kvstore started crashing. I see the following in mongod.log:
2018-01-12T19:02:34.677Z W CONTROL No SSL certificate validation can be performed since no CA file has been provided; please
specify an sslCAFile parameter
The server.conf on both machines points to the same CA cert. I've confirmed the CA certs on both machines have the same md5 hash and permissions.
I also see this in the mongod.log on the problem indexer:
2018-01-12T19:02:34.694Z I CONTROL [initandlisten] options: { net: { port: 8191, ssl: { PEMKeyFile: "/opt/splunk/etc/auth/mycerts/index01_cert.pem", PEMKeyPassword: "", allowInvalidHostnames: true, disabledProtocols: "noTLS1_0,noTLS1_1", mode: "requireSSL", sslCipherConfig: "ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RS..." }, unixDomainSocket: { enabled: false } }, replication: { oplogSizeMB: 200, replSet: "DE599A03-4B9A-426B-BDE9-882044E6E8C3" }, security: { javascriptEnabled: false, keyFile: "/opt/splunk/var/lib/splunk/kvstore/mongo/splunk.key" }, setParameter: { enableLocalhostAuthBypass: "0" }, storage: { dbPath: "/opt/splunk/var/lib/splunk/kvstore/mongo", mmapv1: { smallFiles: true } }, systemLog: { timeStampFormat: "iso8601-utc" } }
From what I see in the server.conf.spec, all of the [kvstore] SSL options, like caCertFile and caCertPath, are deprecated.
Identical configs, identical certs... Why is mongodb having issues on only one machine?
