I have ITSI 3.0, and on a regular basis it is reporting a kvstore connection test.
The problem is that the check triggers a false positive error message in splunkd.log internal logs
In ITSI 2.* the message was like
05-09-2017 06:04:18.605 -0400 ERROR HttpListener - Exception while processing request from 127.0.0.1 for /servicesNS/nobody/SA-ITOA/storage/collections/data/dummy_collection_nvfjdnvjkfdnvjkfnvjkfnvernvjfnvjkfsdnvuenvkjfnvjka?output_mode=json: Could not find object id=dummy_collection_nvfjdnvjkfdnvjkfnvjkfnvernvjfnvjkfsdnvuenvkjfnvjka
In ITSI 3.0, it now looks like
01-23-2018 13:00:01.622 -0800 ERROR HttpListener - Exception while processing request from 127.0.0.1 for /servicesNS/nobody/SA-ITOA/storage/collections/data/this_is_a_kvstore_heartbeat_it_is_not_an_error_please_ignore?output_mode=json: Could not find object id=this_is_a_kvstore_heartbeat_it_is_not_an_error_please_ignore
Can I find a way to drop those event?
I do not want to index them.
The long term solution will be to wait for a future version of Splunk or ITSI that does not generate that log.
A possible workaround now is to setup a nullQueue filter on the search-head indexing the logs to drop them at index time.
PS : This will not prevent the logs to be in the splunkd.log file on disk, just to drop them at index time.
Method to setup a nullQueue filter on the search-head :
[ITSInullqueuefalsepositive] REGEX = (dummy_collection_nvfjdnvjkfdnvjkfnvjkfnvernvjfnvjkfsdnvuenvkjfnvjka|this_is_a_kvstore_heartbeat_it_is_not_an_error_please_ignore) DEST_KEY = queue FORMAT = nullQueue # to delete false positive HTTPlisterner ITSI events.
I understand this is fixed as part of ITSI 3.1. It's not in the release notes although I can confirm that I'm not seeing this anymore.
I confirm, it was fixed in ITSI since version 3.1.0
it will be added to the release notes (as ITOA-8623)