Splunk Search

How do I fix DMC KVstore "Collection Metrics" error

coreyCLI
Path Finder

I recently added a new SH to our SHC.  Show shcluster-status is good, show kvstore-status is good.  I created some kvstore entries on SH1 and they replicated to SH5 (the new one).  I also see the results of scheduled searches that are writting to the kvstore(s) being replicated to the new SH.   I see no errors in splunkd.log.  However, when ever the DMC tries to access /services/server/introspection/kvstore/collectionstats for the "KV Store: Instance" panel, it errors with 
"The Rest request on the endpoint URI /services/server/introspection/kvstore/collectionstats?count=0 returned HTTP 'status not OK': code=500 internal server error'"

I can hit the other /kvstore APIs just fine, .../kvstore/replicasetstats and .../kvstore/serverstats.

If i run "| rest splunk_server=local /services/server/introspection/kvstore/collectionstats" on the new search head in a search bar I get the same error.

This is version 8.2.0 and the kvstore is using the wiredTiger storage engine.

Any suggestions?

 

TIA

Labels (1)
0 Karma

Frank84
New Member

How did you fix this? I am currently experiencing the same error after adding a new search head to our cluster.

0 Karma
Get Updates on the Splunk Community!

Splunk Observability Cloud | Unified Identity - Now Available for Existing Splunk ...

Raise your hand if you’ve already forgotten your username or password when logging into an account. (We can’t ...

Index This | How many sides does a circle have?

February 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

Registration for Splunk University is Now Open!

Are you ready for an adventure in learning?   Brace yourselves because Splunk University is back, and it's ...