topic How to find KVStore status in internal logs in Getting Data In

How to find KVStore status in internal logs

santosh_hb — Thu, 22 Nov 2018 09:53:42 GMT

Hi all,

I have got a task where I have to find the KVStore status through Splunk internal logs. I neither have access to Monitoring Console nor I can run the CLI commands. I have only access to Splunk internal logs. Could you please help me with this? thanks, Santosh

Re: How to find KVStore status in internal logs

Richfez — Fri, 23 Nov 2018 03:08:30 GMT

I guess it depends on what you mean by "status".

index=_internal kvstore

Will get you something - metrics, which might tell you enough, and other things too.

I expect you will like this next thing better, though - there's a whole REST thing you can do, if you have those rights. Give it a try and see...

| rest /servicesNS/nobody/MYAPP/storage/collections/config

will show you all the configured collections in app MYAPP, and

| rest /servicesNS/nobody/MYAPP/storage/collections/data/COLLECTIONNAME

Will show you the data, which you could always | stats count or something.

You can use some of the stuff from the REST API docs to do this, but know that you can't edit anything from | rest in Splunk, because security. (Which I'm not sure makes sense since curl can, with the right password? What? Anyway, no worries on that).

Anwyay, those examples can be modified like my examples to do things.

If that's what you needed, great. If it isn't, well, tell us more specifically exactly what it is you need - WHAT status are you looking for? - and maybe we can help!

-Rich

Re: How to find KVStore status in internal logs

santosh_hb — Fri, 23 Nov 2018 09:55:14 GMT

Hi Rich, Thanks for sharing the details. My issue is: I want to fetch the KVStore running status (Success/Failure/...) from multiple hosts.
I need something like below:

Hostname KVStore status

host1 running
host2 failed

thanks,
Santosh

Re: How to find KVStore status in internal logs

gjanders — Sat, 24 Nov 2018 21:04:20 GMT

In Alerts for SplunkAdmins , or github I have:
AllSplunkEnterpriseLevel - KVStore Process Terminated
SearchHeadLevel - Detect MongoDB errors

And/or potentially SearchHeadLevel - KVStore Or Conf Replication Issues Are Occurring

They might provide detection of when it is crashing, the mongodb checks the logs are still updating which is a sign of it running...