I have a KPI base search which is:
I have 2 or 3 services with KPIs based on this base search. each has Matched entities which are a subset of the entities returned by the base search
Service A correctly filters to its Matched Entities
Service B filters to its matched entities, PLUS the matched entities from Service A!
if I delete the KPI from Service A and delete and recreate the KPI on Service B, Service B will now correctly show only its matched entities
If I re-add the KPI to Service A, Service B will again show both its Matched Entities PLUS those from Service A
Does anyone have any idea what might be going wrong and how to fix it?
PS. The PerEntity Threshold screen on the Configure Service page for each service correctly shows only the matched entities for that service in its preview window
EDIT - Partial solution
So it seems the problem is caused by Service A's entity match rules.
All services had a simple Entity Title matches *servicename*
Service A has an extra AND does not match *_excludethis
When I change the matching rules on Service A so that the extra rule is removed, the other services Dont pick up on Service A's Matched Entities
Note, the other services were NOT picking up on entities that simply matched the extra rule, they picked up Service A's entities
This behaviour seems like a bug to me, surely one service's setup shouldn't affect other services? I still want to know the underlying cause.
This issue was fixed in version 4.0.4 (https://docs.splunk.com/Documentation/ITSI/4.0.4/ReleaseNotes/Fixedissues#Service_Definition).
Note that once you upgrade to a version with the fix, you need to run the kvstoretojson mode 4 option. See the version-specific upgrade notes for 4.0.4 here: https://docs.splunk.com/Documentation/ITSI/4.4.0/Install/VersionNotes
As a workaround for pre-4.0.4 versions, you can open the service and go to the Entities tab. If you see an entity with a "does not match" rule, remove the rule and find a way to match to an entity differently that does not use the “does not match” rule.