Splunk ITSI - using different types of entities in service

[abarneb]

I am filtering a service using two types of entities; one that gives information about hosts and one that gives information about procesess. I've created a KPI that outputs information about processes running on hosts and want to use the entities in service to filter the results. Does anyone have any examples/documentation on how to do this? Is it possible to specify both host and process in the "Entity Split by field" in the KPI definition? I've tried to add "process,host", but this does not seem to work.

These are the fields generated from the search:
process, subsystem, service, server_role, host

Entity Split by field: process
Data filtered by service entities in field: process
Service matches entities on fields: process, subsystem, service, server_role, host

The columns in the entities are of type alias and match the output of the KPI search.

[spandana]

Hi abarneb ,

I'm also facing similar issue. Used process_host as entity but the downside you mentioned is valid. Did you try any other approach which worked for this issue?

[akira_splunk]

Here's a possible solution to match on multiple fields:
1. Import entities from a search and create a new field in the search (e.g. index=blah | eval server_process=host+"."+process | dedup server_process | table server_process, host, process)
2. Create different KPI base search for the process metrics and leverage the field you defined in the entity import (e.g. index=blah | eval server_process=host+"."+process | table server_process, host, process, metric).
3. When defining the KPI in the service, reference the title matching field as server_process

[paulstout · Accepted solution]

Hi there!

I work on ITSI as well as from my understanding you can only have one entity split by field when configuring the KPI. I have a few situations where I have entities as "hosts" and entities as processors/queues within Splunk (name=x, group=x, etc) but do not have KPIs that reference both a host and a process.

I checked up on the documentation and all of the language when describing the entity lookup field is singular and does not mention multiple split-by fields:

http://docs.splunk.com/Documentation/ITSI/latest/Configure/HowtocreateKPIsearches#Split_by_Entity

Whereas in correlation searches, the documentation specifies that multiple fields can be used for throttling:

http://docs.splunk.com/Documentation/ITSI/latest/User/CreateCorrelationSearch

What are you trying to monitor by splitting your KPI by process and entity? Although it doesn't seem possible to configure this in the ITSI KPI search, there may be other ways to achieve your monitoring goals.

[abarneb]

Hi and thanks for your feedback 🙂

I would like to use a base search that outputs a count per process and server and then set up KPIs for each process where I use this base search and filter processes and environment (prod, QA, test servers) by the entities in the service. If I split by the host field, I will get all processes running on the host. If I use processes as entities I will get results from multiple servers belonging to different environments. I could get around this if I didn't use base search. I tested creating entitities on process_host and set up a base search with a process_host field and this works. I was thinking that it is good to use a base search in terms of performance. The downside is that I will need to create/maintain these entities. If you have any thoughts on this approach, that would be greatly appreciated 🙂