Splunk IT Service Intelligence

ITSI Entity Filtering includes entities from other services

Path Finder

I have a KPI base search which is:

  • split by entity,
  • has an entity split field which is an eval of the host + the service name,
  • is set to filter by entities in service
  • and has the entity filter field set as the same entity split field
  • the entity alias filtering field is blank.

I have 2 or 3 services with KPIs based on this base search. each has Matched entities which are a subset of the entities returned by the base search

Service A correctly filters to its Matched Entities
Service B filters to its matched entities, PLUS the matched entities from Service A!

if I delete the KPI from Service A and delete and recreate the KPI on Service B, Service B will now correctly show only its matched entities

If I re-add the KPI to Service A, Service B will again show both its Matched Entities PLUS those from Service A

Does anyone have any idea what might be going wrong and how to fix it?

PS. The PerEntity Threshold screen on the Configure Service page for each service correctly shows only the matched entities for that service in its preview window

EDIT - Partial solution

So it seems the problem is caused by Service A's entity match rules.

All services had a simple Entity Title matches *servicename*

Service A has an extra AND does not match *_excludethis

When I change the matching rules on Service A so that the extra rule is removed, the other services Dont pick up on Service A's Matched Entities

Note, the other services were NOT picking up on entities that simply matched the extra rule, they picked up Service A's entities

This behaviour seems like a bug to me, surely one service's setup shouldn't affect other services? I still want to know the underlying cause.

0 Karma

Splunk Employee
Splunk Employee

This issue was fixed in version 4.0.4 (https://docs.splunk.com/Documentation/ITSI/4.0.4/ReleaseNotes/Fixedissues#Service_Definition).

Note that once you upgrade to a version with the fix, you need to run the kvstore_to_json mode 4 option. See the version-specific upgrade notes for 4.0.4 here: https://docs.splunk.com/Documentation/ITSI/4.4.0/Install/VersionNotes

As a workaround for pre-4.0.4 versions, you can open the service and go to the Entities tab. If you see an entity with a "does not match" rule, remove the rule and find a way to match to an entity differently that does not use the “does not match” rule.

Path Finder

is this issue ITSI-1868? I think we are on 4.0.2

0 Karma
Get Updates on the Splunk Community!

Routing logs with Splunk OTel Collector for Kubernetes

The Splunk Distribution of the OpenTelemetry (OTel) Collector is a product that provides a way to ingest ...

Welcome to the Splunk Community!

(view in My Videos) We're so glad you're here! The Splunk Community is place to connect, learn, give back, and ...

Tech Talk | Elevating Digital Service Excellence: The Synergy of Splunk RUM & APM

Elevating Digital Service Excellence: The Synergy of Real User Monitoring and Application Performance ...