Splunk IT Service Intelligence

ITSI is generating noisy false positives "this_is_a_kvstore_heartbeat_it_is_not_an_error_please_ignore", How to filter them out

mataharry
Communicator

I have ITSI 3.0, and on a regular basis it is reporting a kvstore connection test.
The problem is that the check triggers a false positive error message in splunkd.log internal logs

In ITSI 2.* the message was like

  05-09-2017 06:04:18.605 -0400 ERROR HttpListener - Exception while processing request from 127.0.0.1 for /servicesNS/nobody/SA-ITOA/storage/collections/data/dummy_collection_nvfjdnvjkfdnvjkfnvjkfnvernvjfnvjkfsdnvuenvkjfnvjka?output_mode=json: Could not find object id=dummy_collection_nvfjdnvjkfdnvjkfnvjkfnvernvjfnvjkfsdnvuenvkjfnvjka

In ITSI 3.0, it now looks like

      01-23-2018 13:00:01.622 -0800 ERROR HttpListener - Exception while processing request from 127.0.0.1 for /servicesNS/nobody/SA-ITOA/storage/collections/data/this_is_a_kvstore_heartbeat_it_is_not_an_error_please_ignore?output_mode=json: Could not find object id=this_is_a_kvstore_heartbeat_it_is_not_an_error_please_ignore 

Can I find a way to drop those event?
I do not want to index them.

Tags (2)
1 Solution

yannK
Splunk Employee
Splunk Employee

The long term solution will be to wait for a future version of Splunk or ITSI that does not generate that log.
A possible workaround now is to setup a nullQueue filter on the search-head indexing the logs to drop them at index time.
see http://docs.splunk.com/Documentation/SplunkCloud/6.6.3/Forwarding/Routeandfilterdatad#Discard_specif...

PS : This will not prevent the logs to be in the splunkd.log file on disk, just to drop them at index time.

Method to setup a nullQueue filter on the search-head :

in transforms.conf

[splunkd]
TRANSFORMS-ITSInullqueue=ITSInullqueuefalsepositive

in props.conf

[ITSInullqueuefalsepositive]
REGEX = (dummy_collection_nvfjdnvjkfdnvjkfnvjkfnvernvjfnvjkfsdnvuenvkjfnvjka|this_is_a_kvstore_heartbeat_it_is_not_an_error_please_ignore)
DEST_KEY = queue
FORMAT = nullQueue
 # to delete false positive HTTPlisterner ITSI events.

View solution in original post

0 Karma

sloshburch
Splunk Employee
Splunk Employee

I understand this is fixed as part of ITSI 3.1. It's not in the release notes although I can confirm that I'm not seeing this anymore.

yannK
Splunk Employee
Splunk Employee

I confirm, it was fixed in ITSI since version 3.1.0
it will be added to the release notes (as ITOA-8623)

yannK
Splunk Employee
Splunk Employee

The long term solution will be to wait for a future version of Splunk or ITSI that does not generate that log.
A possible workaround now is to setup a nullQueue filter on the search-head indexing the logs to drop them at index time.
see http://docs.splunk.com/Documentation/SplunkCloud/6.6.3/Forwarding/Routeandfilterdatad#Discard_specif...

PS : This will not prevent the logs to be in the splunkd.log file on disk, just to drop them at index time.

Method to setup a nullQueue filter on the search-head :

in transforms.conf

[splunkd]
TRANSFORMS-ITSInullqueue=ITSInullqueuefalsepositive

in props.conf

[ITSInullqueuefalsepositive]
REGEX = (dummy_collection_nvfjdnvjkfdnvjkfnvjkfnvernvjfnvjkfsdnvuenvkjfnvjka|this_is_a_kvstore_heartbeat_it_is_not_an_error_please_ignore)
DEST_KEY = queue
FORMAT = nullQueue
 # to delete false positive HTTPlisterner ITSI events.
0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...