topic Re: ITSI is generating noisy false positives "this_is_a_kvstore_heartbeat_it_is_not_an_error_please_ignore", How to filter them out in Splunk ITSI

ITSI is generating noisy false positives "this_is_a_kvstore_heartbeat_it_is_not_an_error_please_ignore", How to filter them out

mataharry — Tue, 23 Jan 2018 21:13:13 GMT

I have ITSI 3.0, and on a regular basis it is reporting a kvstore connection test.
The problem is that the check triggers a false positive error message in splunkd.log internal logs

In ITSI 2.* the message was like

  05-09-2017 06:04:18.605 -0400 ERROR HttpListener - Exception while processing request from 127.0.0.1 for /servicesNS/nobody/SA-ITOA/storage/collections/data/dummy_collection_nvfjdnvjkfdnvjkfnvjkfnvernvjfnvjkfsdnvuenvkjfnvjka?output_mode=json: Could not find object id=dummy_collection_nvfjdnvjkfdnvjkfnvjkfnvernvjfnvjkfsdnvuenvkjfnvjka

In ITSI 3.0, it now looks like

      01-23-2018 13:00:01.622 -0800 ERROR HttpListener - Exception while processing request from 127.0.0.1 for /servicesNS/nobody/SA-ITOA/storage/collections/data/this_is_a_kvstore_heartbeat_it_is_not_an_error_please_ignore?output_mode=json: Could not find object id=this_is_a_kvstore_heartbeat_it_is_not_an_error_please_ignore

Can I find a way to drop those event?
I do not want to index them.

Re: ITSI is generating noisy false positives "this_is_a_kvstore_heartbeat_it_is_not_an_error_please_ignore", How to filter them out

yannK — Tue, 23 Jan 2018 21:26:55 GMT

The long term solution will be to wait for a future version of Splunk or ITSI that does not generate that log.
A possible workaround now is to setup a nullQueue filter on the search-head indexing the logs to drop them at index time.
see http://docs.splunk.com/Documentation/SplunkCloud/6.6.3/Forwarding/Routeandfilterdatad#Discard_specific_events_and_keep_the_rest

PS : This will not prevent the logs to be in the splunkd.log file on disk, just to drop them at index time.

Method to setup a nullQueue filter on the search-head :

in transforms.conf

[splunkd]
TRANSFORMS-ITSInullqueue=ITSInullqueuefalsepositive

in props.conf

[ITSInullqueuefalsepositive]
REGEX = (dummy_collection_nvfjdnvjkfdnvjkfnvjkfnvernvjfnvjkfsdnvuenvkjfnvjka|this_is_a_kvstore_heartbeat_it_is_not_an_error_please_ignore)
DEST_KEY = queue
FORMAT = nullQueue
 # to delete false positive HTTPlisterner ITSI events.

Re: ITSI is generating noisy false positives "this_is_a_kvstore_heartbeat_it_is_not_an_error_please_ignore", How to filter them out

sloshburch — Thu, 24 May 2018 21:02:28 GMT

I understand this is fixed as part of ITSI 3.1. It's not in the release notes although I can confirm that I'm not seeing this anymore.

Re: ITSI is generating noisy false positives "this_is_a_kvstore_heartbeat_it_is_not_an_error_please_ignore", How to filter them out

yannK — Thu, 24 May 2018 22:10:21 GMT

I confirm, it was fixed in ITSI since version 3.1.0
it will be added to the release notes (as ITOA-8623)