I have a Splunk 6.4 environment with 3-member SH Cluster running kvstore without replication to the indexer tier.
The kvstore is not particularly heavily utilised, with only three user-defined collections. The biggest of these is a table with ~130,000 rows, while the other two are both <30,000 rows.
(the Cluster also runs Enterprise Security with some vendor apps installed for good measure. Between them, these also defined some collections, but their contribution is negligible - fewer than 10,000 rows in total)
All three lookups operate as state tables - they are frequently updated, with new data being written and old existing data deleted from them, and I suspect this could be a cause of the problem I'm seeing, which is the total size of the MongoDB files in,
$SPLUNK_HOME/var/lib/splunk/kvstore/mongo
is,
SH1 - 13GB
SH2 - 2.8GB
SH3 - 12GB
The 2.8GB on SH2 looks almost plausible for the amount of data I have in my lookups, but the >10GB sizes on the other two SH's.. no way.
Checking operation of kvstore on each SHC member using,
curl -k -u https://localhost:8089/services/server/introspection/kvstore/serverstatus
returns (albeit fairly incomprehensible) introspection data, so the kvstore on the two bloated SHC Members shouldn't be stale and wouldn't benefit from a resync... or would it?
Thanks, folks.
... View more