splunk showAll kvstore members had status starting and and mongod.log shows the messages like below:
2018-11-10T13:38:56.722Z I CONTROL [initandlisten] db version v3.4.10-splunk
2018-11-10T13:38:56.722Z I CONTROL [initandlisten] git version: 078f28920cb24de0dd479b5ea6c66c644f6326e9
2018-11-10T13:38:56.722Z I CONTROL [initandlisten] OpenSSL version: OpenSSL 1.0.2o-fips 27 Mar 2018
2018-11-10T11:28:53.101Z E REPL [replExecDBWorker-1] Locally stored replica set configuration is invalid; See http://www.mongodb.org/dochub/core/recover-replica-set-from-invalid-config for information on how to recover from this. Got "BadValue: priority must be 0 when non-voting (votes:0)" while validating { _id: "splunkrs", version: 18094, members: [ { _id: 108, host: "YYYYY136.sXX:9191", arbiterOnly: false, buildIndexes: true, hidden: false, priority: 1.0, tags: { instance: "1C1326B5-875F-49CC-9A87-487378583D36", all: "all" }, slaveDelay: 0, votes: 1 }, { _id: 112, host: "YYYYY131.sXX:9191", arbiterOnly: false, buildIndexes: true, hidden: false, priority: 1.0, tags: { instance: "26F4C711-DF5A-4032-AA3E-00C8894B92CF", all: "all" }, slaveDelay: 0, votes: 1 }, { _id: 96, host: "psin12p231.sXX:9191", arbiterOnly: false, buildIndexes: true, hidden: false, priority: 1.0, tags: { instance: "44147E6E-E1DC-4860-9DEF-FE87340E8114", all: "all" }, slaveDelay: 0, votes: 1 }, { _id: 117, host: "YYYYY132.sXX:9191", arbiterOnly: false, buildIndexes: true, hidden: false, priority: 1.0, tags: { instance: "52BC02A8-B5D7-4E77-83FA-BE48D120FC8B", all: "all" }, slaveDelay: 0, votes: 1 }, { _id: 116, host: "YYYYY138.sXX:9191", arbiterOnly: false, buildIndexes: true, hidden: false, priority: 1.0, tags: { instance: "70EB0B2A-A167-45EC-96AC-4BCFD2F7DBB9", all: "all" }, slaveDelay: 0, votes: 1 }, { _id: 119, host: "YYYYY130.sXX:9191", arbiterOnly: false, buildIndexes: true, hidden: false, priority: 1.0, tags: { instance: "D0395ED0-9258-4C9B-81E9-9F2E7092661D", all: "all" }, slaveDelay: 0, votes: 0 }, { _id: 118, host: "YYYYY133.sXX:9191", arbiterOnly: false, buildIndexes: true, hidden: false, priority: 1.0, tags: { instance: "EA864803-E37A-4229-899E-46895CC98AE8", all: "all" }, slaveDelay: 0, votes: 0 }, { _id: 115, host: "YYYYY137.sXX:9191", arbiterOnly: false, buildIndexes: true, hidden: false, priority: 1.0, tags: { instance: "EBC301CE-1E29-4D4B-B980-210762DFE089", all: "all" }, slaveDelay: 0, votes: 1 }, { _id: 101, host: "psin12p145.sXX:9191", arbiterOnly: false, buildIndexes: true, hidden: false, priority: 1.0, tags: { instance: "FC177089-607F-4E4A-AB65-2A025DE153AB", all: "all" }, slaveDelay: 0, votes: 1 } ], settings: { chainingAllowed: true, heartbeatIntervalMillis: 2000, heartbeatTimeoutSecs: 10, electionTimeoutMillis: 10000, catchUpTimeoutMillis: 60000, getLastErrorModes: {}, getLastErrorDefaults: { w: 1, wtimeout: 0 } } }
