My bad 🙂 I removed Splunk and just rebuilt it fresh. Still same issue, kvstore has been in starting status. This is what I see in mongod.log: 2025-05-29T03:22:15.751Z I CONTROL [LogicalSessionCacheReap] Failed to reap transaction table: NotYetInitialized: Replication has not yet been configured 2025-05-29T03:22:15.747Z I CONTROL [LogicalSessionCacheRefresh] Sessions collection is not set up; waiting until next sessions refresh interval: Replication has not yet been configured   ... View more

It's the existing SHC, just trying to add new member. Captain is working and 8191 port is also opened. This is what I see in mongod log: 2025-05-29T03:22:15.751Z I CONTROL [LogicalSessionCacheReap] Failed to reap transaction table: NotYetInitialized: Replication has not yet been configured 2025-05-29T03:22:15.747Z I CONTROL [LogicalSessionCacheRefresh] Sessions collection is not set up; waiting until next sessions refresh interval: Replication has not yet been configured ... View more

See these events in splunkd: 05-22-2025 21:07:58.608 -0400 ERROR KVStoreAdminHandler [1848035 TcpChannelThread] - An error occurred. 05-22-2025 21:07:36.668 -0400 ERROR KVStoreIntrospection [1848033 TcpChannelThread] - failed to get introspection data 05-22-2025 21:07:19.587 -0400 WARN KVStoreConfigurationProvider [1845927 MainThread] - Action scheduled, but event loop is not ready yet Tried cleaning up kvstore by running "splunk clean kvstore --local --answer-yes" but that didn't change anything, status is still stuck in starting. ... View more

I followed these steps to add new box to existing SHC, everything looks fine on the SHC side but kvstore status has been 'status : starting' since, and it's been over an hour. Is this normal or I missed something? ... View more

Thanks for this answer, using it I was able to resolve my issue of finding out the version of all the Enterprise and UFs my team is responsible to upgrade: index="_internal" source="*metrics.log*" group=tcpin_connections hostname IN (<your.servernames>) | stats latest(fwdType) latest(version) latest(os) by hostname   ... View more

Thanks. That helped me resolving my issue, just a small correction that it would be: target="_blank" ... View more

Did you ever get the answer why clients were becoming members of defaultgroup? ... View more