usually, the problem with long list of entities filters, are long search terms, or scripts timing out when waiting for the REST call to return the list. But if you reach that point, you will be working with a support engineer to tune splunk. ... View more

You have the good approach stats command to measure the first(_time) and last(_time) to then measure the duration or use the transaction command https://docs.splunk.com/Documentation/Splunk/latest/SearchReference/Transaction ... View more

If you were do to a regular search, I would recommend to use a transaction command (staring at the first failure, ending at the latest OR success) Then exclude the transactions with success, and focus on the others, see https://docs.splunk.com/Documentation/Splunk/latest/SearchReference/Transaction But those commands are not great for a KPI search in ITSI, as they are transforming the data ... View more

Entities are an abstract layer to identify an asset. By example an entity could be as basic as a host, but could also be used for a cpu core#, or an application on a server ... An entity is defined by alias fields (unique fields values, like a host or a vm id), or info fields (can be the same for several entities, like a datacenter location, a service role ...) in ITSI the entities are used for 2 things : - group entities in a service, using a filter, or a direct link. - for the KPIs in a service in KPI : - you can ask to filter to only the entities in the service, or not (optional) - you can also ask do to a split by of the metrics, to get the detail per entity. (optional) For the split by - if you are use a field (alias/info) to do the entity split by, then it will refer to a real entity - but you could also use a split by field that is not specific to a real entity, we will then say that you are creating "pseudo entities", that only exits in the KPI metrics results (by example do a split by process when you do not use this field for entities) ... View more

You can use a sub search to call the lookup and return an OR condition , to use as a search condition. <my search> [ | inputlookup mylookup | table A ] will become <mysearch> A=x OR A=y OR A=z for more complex conditions, look at the command format https://docs.splunk.com/Documentation/Splunk/latest/SearchReference/Format ... View more

It's a custom command, (commands.conf) calling a python script, which code is in clear in the app SA-ITOA/bin ... ... View more

True, the thresholds per entity are not yet possible. If you have a support contract, do not hesitate to open an Enhancement Request. ... View more

I would put my money on the distinction between Inputs field OR index time fields inputs are like indexed_extractions and meta fields and indextime are on the pipeline on the heavy forwarder and indexers from the typing processor As they are in props/transforms, so the fields names have to be simple without space. A few years ago, with json, the "xxx.yyy" for "aaa:bbb:cc" fields name format became possible search time fields those are more flexible, as you can use 'my space string field' in SPL to call them. ... View more

Not sure what is doable. but if you are going the service-dependencies road, look at the way the service-health scores are calculated https://docs.splunk.com/Documentation/ITSI/4.4.0/Configure/HealthScores the weights can be a way to assign importance to your sub services But there is no way to do something like "2 of 3 are up" with the weights. For services dependencies, the upper service can only look at the last minute sub-services scores, (the current ones are not yet calculated), so there is a cascade effect, and a minute delay to see the updates. ... View more

Is your search returning more than 1 event when it runs ? If it does, maybe massage your events, like using a "|dedup " or "| head 1" to trim them before the notables are created. If your events results are unique but get indexed twice check the _indextime of the notable events, to figure when they were created. Check if you have a useack=true enable in the outputs.conf of your search-head (it can cause the forwarder to attempt to send the same events multiple time in case of network failure) check if you are not cloning your data to 2 sets of indexers ... View more

if the timerange windows are still ending up at the same time, then the displayed kpi values and latest service health scores are likely to be identical. (last 4h our last 1 hour, will both show the same last values) but if you changed the end window, let's say from "last 4h" to "yesterday", we should expect the KPI values to change. ... View more

If you make your KPIs Shared base searches (SBS) generic enough, and put the entity logic in the services entity filters, then you can rely on the service's templates + SBS to scale ideally, the service template use SBS, the SBS are broad and use the "filter per entity per service" then when you create services (or import entities in service), the KPIs will automatically run on the new list of entities. Beware, the usual limit of entity cardinality is 10000, so if you have a SBS populating several KPI metrics, for many entities, (number of metrics * number of entities > 10000), you may want to split it in several SBS ... View more

2 remarks : 348 seconds of indexing delay, this is not great. you want to be under a minute. Maybe your servers clocks are drifting ... When you run the KPI searches do you get results, this will be a good test. Are your KPIs linked to Shared bases searches (SBS)? Do you use a split by entity, how many entities total would be indirectly linked to a single SBS ? (10000 is usually the cadinality limit by default) Try to convert to a hadhoc KPI in a single service to compare. ... View more

It sounds like a Microsoft admin question. Those data seem to be from a windows forwarder monitoring, but microsoft explains that the number depends of the number of cores. Microsoft® Description – Processor Queue Length is the number of threads in the processor queue. Unlike the disk counters, this counter counters, this counter shows ready threads only, not threads that are running. There is a single queue for processor time even on computers with multiple processors. Therefore, if a computer has multiple processors, you need to divide this value by the number of processors servicing the workload. A sustained processor queue of less than 10 threads per processor is normally acceptable, dependent of the workload. Maybe you could use an more complex KPI search to also compare the number of cores, and normalize your metric. ... View more

some licenses can be in their own license stack, so you could check that. some licenses are specific to some sourcetypes (the new ITSI internal is like that). ... View more

This is not surprising. data change on disk the KPI run and update their summary values at best every minute - delay the kpi is indexed the service score is calculated based on the kpis values from the previous minute (as the current minute may not be indexed) + delay the service score is indexed if the service score has a dependency over another service, another minute of delay to wait for those dependent services healthscores + extra delay also if you have indexing/forwarding slowness between the SH and the indexers, add some delay. So it could take 2-3 minutes for the service score to flip. ... View more

Look like it could be a splunk role permission issue. Check if you are member of the "admin" role and of the "itoa-admin" role, and make sure that some of their default capabilities were not removed, Verify that ITSI teams have the proper roles listed under. The other possibility is that the path of $SPLUNK_HOME on the 2 environments are different, and the backup is looking for a folder path that does not exist/is not accessible ? ... View more

ITSI is an application on top of splunk enterprise. if you can send the data to splunk, you can use it in ITSI you can use any type of data collection to import the topology of your deployment, from a search to a csv import, and there are also some automatic detection tools provided. In ITSI we call them entities and services, and their definitions are stored in a kvstore. see https://docs.splunk.com/Documentation/ITSI/latest/Configure/EntitiesOverview ... View more

As KPI are summary searches, there are no use cases for reporting on pooling of more than 24h. if you want to report on data from a larger timerange, you could precalculate it using another scheduled search, and summarize the results (or use a lookup), then carve a custom KPI to retrieve it, and add fake _time fields. ... View more

ITSI KPI searches need to preserve necessary fields to work, As mentioned here: https://docs.splunk.com/Documentation/ITSI/latest/Configure/AddKPIs#Define_a_source_search_from_an_ad_hoc_search The ad hoc search string that you create. This is the event gathering search for the KPI. Note: The use of transforming commands, the mstats command, the gettime macro, or time modifiers in your KPI search is not recommended as this may cause issues with KPI backfill, the display of raw data on ITSI views such as glass tables and deep dives that allow you to run KPI searches against raw data, and the KPI threshold preview. in your case, you need to preserve the field _time. (maybe others, depending of your filters/entities lookup mapping etc...) try to add something like that to your stats command | stats first(_time) AS _time and to your fields | fields _time ... ... View more

The field "event_id" is mostly used for the notable events, in the itsi_tracked_alerts field. but in versions older than 3.* it was also used for the group id in the index=itsi_grouped_alerts, and since 4.* for the Episodes ids too. This can be confusions, as there is also an itsi_group_id field and when you use the API, the id may be referring to the notable or to the group... The best way to avoid confusion is to do an eval or a rename to create a second field group_id to avoid confusion, depending of the context of the data. ... View more

The reason why your KPI is not summarizing events is likely that your search is removing critical fields from the results. Transformation commands are not allowed in a KPI. see https://docs.splunk.com/Documentation/ITSI/latest/Configure/AddKPIs#Define_a_source_search_from_an_ad_hoc_search The ad hoc search string that you create. This is the event gathering search for the KPI. Note: The use of transforming commands, the mstats command, the gettime macro, or time modifiers in your KPI search is not recommended as this may cause issues with KPI backfill, the display of raw data on ITSI views such as glass tables and deep dives that allow you to run KPI searches against raw data, and the KPI threshold preview. in your comment your search was : ("FAIL*" Connection timed out sourcetype="XXXXXX") date_minute="*" earliest=-2mon@mon latest=now| top limit=20 date_hour The "top" command will remove all fields, except the "date_hour" ITSI KPI needs to have at least the field "_time" preserved in your results to be able to do some calculations. Also depending on the fields you are using for the KPI service aggregate, the entity calculation, and the entity filter, and the entity split by, you have to make sure that those fields are preserved. ... View more