ITSI KPI - Splitting by multiple entities.

[eddieddieddie]

I am collecting data which tests that a server can reach other destinations. This data is collected in the form of source, destination, application name, description and state - did it connect or not. I would like to use this data in a ITSI Service's KPI in to measure the state of connections from applications to other servers. However in ITSI it seems only possibly to split by one value not two, but I read that it's possibly to create new field combining the two fields together to get around this. I've tried this but I'm getting some unexpected results:

I created a base search which creates a new combined field called 'host_application':

index=main task=checkopenports | eval host_application = host . "_" . application

I added a new entity to the service called host_application and set it to match host123_websiteservice (which is what I want to pick out of the results to match for this service: the server 'host123' and the application 'websiteservice')

In the KPI configuration I configured 'Entity Split Field' to host_application and also set 'Entity Filter Field' to host_application, and then set the calculation to 'count'

I'm hoping this should split the results by the host_application field and then only show items matching the host_application field set in the entities tab but instead it shows a value of 0 and no entities. Any suggestions as to what I'm doing wrong here?

Thanks

Eddie

[eddieddieddie]

Thanks for the reply and insights. I didn't know that the 'Entity Split Field' doesn't have to reference an entity defined in the service - it can just reference any field you know will be in your data.

However one issue still continues to bug me and I can't quite work out a solution: as I understand it when creating a KPI in a service if 'Filter to Entities in Service' is enabled the 'Entity Filter Field' is used to filter the data to only that which contains both the Entity field name and the value you associated with it as an alias in the entities of the service.

I find that this only works for me if the Entity Filter Field of 'host' is used, anything else I use never matches anything (so the KPI is always null). Does the Entity Filter Field need to be declared somewhere else in the service maybe? Or does this field have restrictions like it's case sensitive or can't contain not alphanumeric characters?

Thanks

For reference: I am doing this in ITIS version 4.4.5 in Splunk Enterprise version 8.0.6.

[eduncan · Accepted solution]

Actually, even if you have Entity Filter field on, that Entity split by field can be just a field in the data.  It doesn't look at that field at all in regards to an entity in your entity table.  If you can't get entities to filter if you have another field other than host, you need to make sure that the field you are wanting to filter by (example ComputerName) is actually in the data you are getting.  Open your KPI in the Service, click the edit button by the KPI, on the Bottom, expand that Generated search and click anywhere in the white space to open that search in a Search window.  Run it.  If you get no events back (other than something that says N/A), take everything off the search that is appended by the KPI (should start with | `aggregate_raw_).  Take all that off and just run the base search.  Look at the event fields that come back and verify that the field name you have in that Filter by Entities box, actually exists.  98% of the time the reason that you won't get entities to filter properly is that you are trying to filter by a field that doesn't exist in the data.  Also if you are doing that regex in your KPI search, you need to instead do an extraction so that the field exists when the KPI runs.  Best practice...don't ever do regex inside of KPIs, do extractions.  Hope that helps.

[eddieddieddie]

Again thanks for the informative reply.

As suggested I opened the generated search and removed everything after '| `aggregate_raw_'. This I found was returning no results at all. So I experimented further with parts of that search query, which started out like:

eventtype=microsoft_iis_web
  | eval 4xx_error=if(status>=400 AND status<500, status, null()), 5xx_error=if(status>=500 AND status<600, status, null())
  | search
    [| rest splunk_server=local report_as=text "/servicesNS/nobody/SA-ITOA/itoa_interface/generate_entity_filter?service_id=1ae7cab6-424e-4d10-be71-60800c732a12&kpi_id=fc5067c63e829692600055c1&entity_id_fields=s_sitename&entity_alias_filtering_fields=s_sitename&search_type=adhoc"
    | return $value ]

From here I took just the rest query and experimented with the entity_alias_filtering_fields argument, starting with:

| rest splunk_server=local report_as=text "/servicesNS/nobody/SA-ITOA/itoa_interface/generate_entity_filter?service_id=1ae7cab6-424e-4d10-be71-60800c732a12&kpi_id=fc5067c63e829692600055c1&entity_id_fields=s_sitename&entity_alias_filtering_fields=s_sitename&search_type=adhoc"

It returns `no_entities_matched`.

However when I changed entity_alias_filtering_fields from "s_sitename" to "host" it returns the value configured for host in the Services entities tab. From here I went on to experiment with other alias's in the Entities section of the service - only the host field ever works, and only for hosts ITSI is aware of (so if I changed the Alias for host to add a hostname that doesn't exist it wouldn't be returned by the above rest query).

But after this I tried something further - I looked up the entity in the 'Entities' list (from the Configure menu in ITSI) and added an alias for what I was hoping to filter the entity by. Suddenly the rest call above started working, and after re-creating the KPI that started working correctly too!

So it seems the cause of this 'fault' was my mis-reading of the documentation - I had been assuming that mentions of 'entities' when creating KPIs refer to what's in the Entities tab of the Service, but in fact refers to an alias configured against the Entity as a whole in ITSI.

[eduncan]

You don't actually need to add this field to your entity or even make it an entity.  The 'split by' field is really just a field in the data that you want to split by for view purposes only.  The entity filter info is looking for data that comes in from the raw data and matches something in your entity table.  Best practice is not to do a combo field but pick something that can always identify that entity in the raw data.  I would switch your entity filter field back to something that is in entity table itself.  If you are trying to show a KPI that only shows certain entities for a service, you still need to make sure that in your entity properties you have something in there from the raw data that can also be matched when the KPI runs.  For instance if I want entities that have a field 'host123' AND 'app123' I need to go to my Entity Filter tab in the Service and add those two things in.  Entity split fields are not used to filter anything but rather to view the data in a diff way.  For instance if I have 404 errors I may want to see them by host and by JVM.  To accomplish this I have one KPI that has 'host' in the entity split field, and another KPI that has the 'uri' in the split by field.  Hope that helps.