Splunk ITSI

ITSI is generating noisy false positives "this_is_a_kvstore_heartbeat_it_is_not_an_error_please_ignore", How to filter them out

mataharry
Communicator

I have ITSI 3.0, and on a regular basis it is reporting a kvstore connection test.
The problem is that the check triggers a false positive error message in splunkd.log internal logs

In ITSI 2.* the message was like

  05-09-2017 06:04:18.605 -0400 ERROR HttpListener - Exception while processing request from 127.0.0.1 for /servicesNS/nobody/SA-ITOA/storage/collections/data/dummy_collection_nvfjdnvjkfdnvjkfnvjkfnvernvjfnvjkfsdnvuenvkjfnvjka?output_mode=json: Could not find object id=dummy_collection_nvfjdnvjkfdnvjkfnvjkfnvernvjfnvjkfsdnvuenvkjfnvjka

In ITSI 3.0, it now looks like

      01-23-2018 13:00:01.622 -0800 ERROR HttpListener - Exception while processing request from 127.0.0.1 for /servicesNS/nobody/SA-ITOA/storage/collections/data/this_is_a_kvstore_heartbeat_it_is_not_an_error_please_ignore?output_mode=json: Could not find object id=this_is_a_kvstore_heartbeat_it_is_not_an_error_please_ignore 

Can I find a way to drop those event?
I do not want to index them.

Tags (2)
1 Solution

yannK
Splunk Employee
Splunk Employee

The long term solution will be to wait for a future version of Splunk or ITSI that does not generate that log.
A possible workaround now is to setup a nullQueue filter on the search-head indexing the logs to drop them at index time.
see http://docs.splunk.com/Documentation/SplunkCloud/6.6.3/Forwarding/Routeandfilterdatad#Discard_specif...

PS : This will not prevent the logs to be in the splunkd.log file on disk, just to drop them at index time.

Method to setup a nullQueue filter on the search-head :

in transforms.conf

[splunkd]
TRANSFORMS-ITSInullqueue=ITSInullqueuefalsepositive

in props.conf

[ITSInullqueuefalsepositive]
REGEX = (dummy_collection_nvfjdnvjkfdnvjkfnvjkfnvernvjfnvjkfsdnvuenvkjfnvjka|this_is_a_kvstore_heartbeat_it_is_not_an_error_please_ignore)
DEST_KEY = queue
FORMAT = nullQueue
 # to delete false positive HTTPlisterner ITSI events.

View solution in original post

0 Karma

sloshburch
Splunk Employee
Splunk Employee

I understand this is fixed as part of ITSI 3.1. It's not in the release notes although I can confirm that I'm not seeing this anymore.

yannK
Splunk Employee
Splunk Employee

I confirm, it was fixed in ITSI since version 3.1.0
it will be added to the release notes (as ITOA-8623)

yannK
Splunk Employee
Splunk Employee

The long term solution will be to wait for a future version of Splunk or ITSI that does not generate that log.
A possible workaround now is to setup a nullQueue filter on the search-head indexing the logs to drop them at index time.
see http://docs.splunk.com/Documentation/SplunkCloud/6.6.3/Forwarding/Routeandfilterdatad#Discard_specif...

PS : This will not prevent the logs to be in the splunkd.log file on disk, just to drop them at index time.

Method to setup a nullQueue filter on the search-head :

in transforms.conf

[splunkd]
TRANSFORMS-ITSInullqueue=ITSInullqueuefalsepositive

in props.conf

[ITSInullqueuefalsepositive]
REGEX = (dummy_collection_nvfjdnvjkfdnvjkfnvjkfnvernvjfnvjkfsdnvuenvkjfnvjka|this_is_a_kvstore_heartbeat_it_is_not_an_error_please_ignore)
DEST_KEY = queue
FORMAT = nullQueue
 # to delete false positive HTTPlisterner ITSI events.
0 Karma
Get Updates on the Splunk Community!

Take Your Breath Away with Splunk Risk-Based Alerting (RBA)

WATCH NOW!The Splunk Guide to Risk-Based Alerting is here to empower your SOC like never before. Join Haylee ...

SignalFlow: What? Why? How?

What is SignalFlow? Splunk Observability Cloud’s analytics engine, SignalFlow, opens up a world of in-depth ...

Federated Search for Amazon S3 | Key Use Cases to Streamline Compliance Workflows

Modern business operations are supported by data compliance. As regulations evolve, organizations must ...