Having the same issue after upgrading from 4.4.5 to 4.7.0.  We just tried going from our broken 4.7.0 to 4.9.0 in hopes that we would get our service analyzer back, instead, we are now getting a license error and ITSI is non-functional.  Not sure what happened after 4.4.5 and Splunk 8.1.5 but we are now down and non-operational.  ... View more

Thanks for this addition information. As you recommended I have changed the entity removing the ‘site’ field from the alias section and recreating it in the ‘Info fields’ section – as below: I then opened the Service and from the Entities Tab, removed ‘site’ as an alias and recreated it as an Info field – so it looks like the following. It does successfully match the correct entity too. In the KPI the ‘entity filter field’ is set to ‘site’. But now the KPI does not work! In the data I can manually search and find ‘[rest of the base search] site=UserInterface’ via search. If I expand the Generated Search for the KPI, open it in search, and reduce the search just back to the base search and the rest lookup I get nothing. So it seems that the generate_entity_filter function isn’t picking up the field 'site' since I changed it to ‘info’. What am I doing wrong here?   ... View more

Are you trying to restrict access to the service view, or the underlying data the search returns?  Metrics have no real private info except a host name so not really sure why you are restricting this way.  Use teams instead from within ITSI to assign which services which members can see. ... View more

Again thanks for the informative reply. As suggested I opened the generated search and removed everything after '| `aggregate_raw_'. This I found was returning no results at all. So I experimented further with parts of that search query, which started out like: eventtype=microsoft_iis_web | eval 4xx_error=if(status>=400 AND status<500, status, null()), 5xx_error=if(status>=500 AND status<600, status, null()) | search [| rest splunk_server=local report_as=text "/servicesNS/nobody/SA-ITOA/itoa_interface/generate_entity_filter?service_id=1ae7cab6-424e-4d10-be71-60800c732a12&kpi_id=fc5067c63e829692600055c1&entity_id_fields=s_sitename&entity_alias_filtering_fields=s_sitename&search_type=adhoc" | return $value ] From here I took just the rest query and experimented with the entity_alias_filtering_fields argument, starting with: | rest splunk_server=local report_as=text "/servicesNS/nobody/SA-ITOA/itoa_interface/generate_entity_filter?service_id=1ae7cab6-424e-4d10-be71-60800c732a12&kpi_id=fc5067c63e829692600055c1&entity_id_fields=s_sitename&entity_alias_filtering_fields=s_sitename&search_type=adhoc" It returns `no_entities_matched`. However when I changed entity_alias_filtering_fields from "s_sitename" to "host" it returns the value configured for host in the Services entities tab. From here I went on to experiment with other alias's in the Entities section of the service - only the host field ever works, and only for hosts ITSI is aware of (so if I changed the Alias for host to add a hostname that doesn't exist it wouldn't be returned by the above rest query). But after this I tried something further - I looked up the entity in the 'Entities' list (from the Configure menu in ITSI) and added an alias for what I was hoping to filter the entity by. Suddenly the rest call above started working, and after re-creating the KPI that started working correctly too! So it seems the cause of this 'fault' was my mis-reading of the documentation - I had been assuming that mentions of 'entities' when creating KPIs refer to what's in the Entities tab of the Service, but in fact refers to an alias configured against the Entity as a whole in ITSI. ... View more

Hey I will load this on my env and take a look and if it is a bug I will let the developer know. ... View more

You are getting multiple episodes because once you get an up or a clear, it breaks and no more NE's can enter that episode.  If you are getting a lot of episodes you may consider using a KPI instead and only create alerts when the KPI turns red. ... View more

I have my “split by entity” option as yes, the field of ‘user_office’ is in the ‘entity split field’ box, this is the same field used in the services entity rules, and the same one used in the entity alias. I have sun entity calculation and the service aggregate is Average, would filling data gaps cause an error? I’ve done this before in another set up and it worked as intended. ... View more

Sure, will try this and update you back. Thanks. ... View more

My email is s.khosrowshahian@gmail.com. ... View more

If you are sure that even in the itsi_summary index that the groupid's for the ones retrieved via rest are NOT there, then I'd open a support case. ... View more

They are timing out because of how many objects are in the KV Store.  Make sure you clean the KV store first and then you can change the default timeout of 12 hours.  The instructions are listed here:https://docs.splunk.com/Documentation/ITSI/4.5.0/Configure/Restore ... View more

When you look in the episode that is created when a KPI is alerting, do you see service_name in the common fields?  When you upgraded, did you clean your KV store first? ... View more

No, I don't, but I replied only because your posting was empty. eduncan has suggested a solution. I suggest you consider it. ... View more

ITSI license is calculated by the events or data written to the itsi summary index.  It doesn't charge for what is ingested in to splunk rather what ITSI is ingesting into it's engine.  You need to look at how much data is being written to the itsi_summary index each day and that is your daily ingest. ... View more

In your actions of your agg policy you need to choose:  if the events in this episode are exactly equal to 1 and that way it creates one episode, not one every time a NE is added.  This way all subsequent NE's are in the episode and linked to ONE Servicenow Incident. ... View more

Are there really 9,000 unique entities that are related to a service?  Make sure that in your adhoc search you are deduping on the host name or entity title name.  If you want to manually add them from a csv, you need to have a field that designates the service they are supposed to be related to.  Best practice is to use something in the actual data of the entity that shows they should be part of a service and NOT a host name because then it is not dynamic.  If you are importing via a search and you have a large number of entities that already exist, it may fail because it is trying to update existing ones.  9K entities is a large number so make sure you are deduping in your ad hoc search. ... View more

I understand groups want their 'stuff' separate from others BUT this doesn't mean you have to create separate services.  Best practice, don't create services based on how people work, create them based on their dependency of the components.  You can still give them a view of their own stuff either through a Glass Table OR you can create views in Service Analyzer, Deep Dives or Episodes.  Personally I like Glass tables because you can create boxes and show whatever you want based on a splunk search.  Instead of dragging over a KPI that may have entities they don't care about, instead write a search pulling from itsi_summary index and filter to the entities you do care about.  This allows them to see just what they care about without you breaking a service apart to accomodate how they work.  Best practice, if you are measuring the same thing on 5 hosts that serve the same purpose (app servers, web servers, databases, or business service) that is one service. For entities and entity filtering you always want to try and filter by something other than a host name.  If you use only a host name or even an alias with the combo of a host and something else, you move towards 'static' membership.  If instead you create enrichment fields that you can then filter by, then membership becomes dynamic.  For instance, if you have a host naming convention on something like this  CHDBBNK3455 where the first two characters are the location (Chicago), the 2nd two are the role (Database), the 3rd are the apps it serves or service (Banking) you can regex those into new fields and create a lookup.  OR if you have info from a cmdb you can create a lookup.  Then you do entity import searches that are scheduled like this:  index=main host=* |lookup myenrichmentdata.csv host ON host.   This is basically saying..."hey lookup this is who I am, what do you know about me?" and all the fields in the lookup become enrichment in your entities.   This way if you wanted to create some view based on something like Support group, you can easily filter by those fields.  They become dynamic instead of static because you schedule it so when a new host starts reporting in, it too will do the lookup, pull it's new info, and then be imported in as an entity.  You never want to have duplicate entites even if the name is different and the underlying 'machine' or CI is the same thing because it will throw of your aggregate health scores.  Hope that helps! ... View more

If you go into settings > searches> choose app ITSI and search for the word entity.  You will see custom entity imports that you have done that you scheduled as recurring.  You can see the actual search that shows entities in Settings>lookups>itsi>itsi_entities.     ... View more

Make sure that in the corr search you have the Notable Event Identifier fields set and not just leaving it at 'source'.  These fields are used to identify the NE as unique.  For instance you might want to use %host%%eventtype%%Message%.  This will let ITSI know that the NE is the exact same one as one already created and it will prevent duplicates. When wanting to create a Remedy ticket you will want to make sure that in the Action tab of the Aggregation policy you choose something like When this event occurs:  Severity greater than or equal to Medium, and then the action will be to create an event.  Agg policies create 1 ticket per episode, not per NE. ... View more

This may not be a bug.  Remember that NE's can make it into multiple episodes on purpose.  If a NE is related to more than one agg policy, it will be grouped with that policy as well.  Make sure that is not the case before thinking it is a bug. ... View more

So first thing to check is that the agg policies you have to group are not excluding events. Skipped events is relevant to grouped events only so when you see skipped events, are they events that are not being grouped by the agg policies.  You can test this by making sure your default policy is grouping by 'source' only.  Also can you tell me if you have tsix enabled?  There is a known issue with large number of NE's being created if you have that enabled.  Troubleshooting, turn off corr searches except for one, and one agg policy.  See if events get grouped and skipped searches reduce.  If so, then you need to analyze what your agg policy is doing that is excluding those events.  Basically it's the process of elimination and remembering skipped events means, skipped grouping, not skipped creating NE's. ... View more