You might want to look over my response in May: https://community.splunk.com/t5/Monitoring-Splunk/Please-share-a-short-Splunk-preventative-tasks-lis...
The should cover KV store issues. ES is reasonably good at posting messages for its own health checks, and that's generally a good way to monitor problems from the monitoring console instance as a scheduled search:
| rest splunk_server=*/servicesNS/-/-/messages
and filter as needed.
Likewise, you can search deployment and splunkd health for colors other than green:
| rest splunk_server=* /services/server/health/deployment/details
| rest splunk_server=* /services/server/health/splunkd/details
The latter search should be transposed to show health by feature.