topic Re: How do I read stored ITSI metrics into a custom dashboard outside of ITSI? in Splunk ITSI

How do I read stored ITSI metrics into a custom dashboard outside of ITSI?

wcooper003 — Wed, 12 Oct 2016 16:52:58 GMT

We're implementing ITSI and we want to import ITSI metrics into a custom dashboard, without having to manually duplicate all of the searches. For instance, we define a service and KPIs in ITSI, then in a separate app/dashboard, we want to read the stored KPI values to present in a dashboard.

The questions are: Where are the ITSI data stored -- is it all in index=itsi_summary? And what is the best way to access the data by KPI and Service name?

I saw a talk at .conf2016 using the query below to access the overall service health score to put in a custom dashboard (for example with a Siteminder service):

index=itsi_summary source=service_health_monitor 
| eval service_name="Siteminder" 
| lookup alarm_console_lookup title as service_name OUTPUT kpis._key as service_id_mv 
| eval service_id = mvindex(service_id_mv, mvfind(service_id_mv, "SHKPI"))

Is this the best approach to do this? Because I seem to get duplicate time values. And what would be the best way to extract the individual KPIs? I think they are in different "source" in the itsi_summary index, but i'll need to map out how to access them by the KPI and service name since they all apparently use hash ids for keys.

I'm hoping someone has done this and it's straight forward without having to spend too much digging through the itsi_summary index.

Thanks

Re: How do I read stored ITSI metrics into a custom dashboard outside of ITSI?

wcooper003 — Tue, 29 Sep 2020 11:29:17 GMT

In case anyone is interested in how to do this, I figure out a solution.

First, I set up a saved search that spits out a kvstore lookup to augment the alarm_console_lookup with the Service and KPI names:

| inputlookup alarm_console_lookup 
    | rename kpis._key as kpis_key, kpis.title as kpis_title, title as Service 
    | eval zipped = mvzip(kpis_key, kpis_title) 
    | mvexpand zipped 
    | fields zipped Service 
    | eval temp=split(zipped,",") 
    | eval key=mvindex(temp,0) 
    | eval KPI=mvindex(temp,1) 
    | fields - temp - zipped 
    | eval source = if(substr(key,1,5)=="SHKPI", "service_health_monitor", "Indicator - ".key." - ITSI Search")
    | table key, KPI, Service, source
    | outputlookup itsi_keys

Then I can use this to quickly pull the metrics I want from itsi_summary:

index=itsi_summary earliest=-241m@m latest=-1m@m 
| lookup itsi_keys key as itsi_kpi_id 
| search Service="Siteminder" 
    AND (KPI=ServiceHealthScore 
    OR (KPI="Authentication Rejections" AND indexed_is_service_max_severity_event=1) 
    OR (KPI="Available Memory (MB)" AND indexed_is_service_max_severity_event=1) )

Seems to work pretty well so far.

Re: How do I read stored ITSI metrics into a custom dashboard outside of ITSI?

r34220 — Mon, 21 Nov 2016 18:56:38 GMT

When I run the first query I get an error "Error in 'outputlookup' command: The lookup table 'itsi_keys' is invalid.". What am i doing wrong 😞

Re: How do I read stored ITSI metrics into a custom dashboard outside of ITSI?

wcooper003 — Tue, 29 Sep 2020 11:54:25 GMT

The outputlookup command it trying to write to a kvstore lookup table that I first defined. In my app, I edited both the collections.conf and transforms.conf in my default folder as follows:

collections.conf:
[itsi_link]

transforms.conf:
[itsi_keys]
external_type=kvstore
collection=itsi_link
fields_list = key, KPI, Service, service_kpi, weight, source

After you define those, the output should work.

Note however: I was just on a call with professional services regarding this topic - and they mentioned that there is a service_kpi_lookup that does something very similar to the itsi_keys lookup I defined above. So you may be able to use that directly in your search.

Re: How do I read stored ITSI metrics into a custom dashboard outside of ITSI?

ebergin1 — Tue, 15 Mar 2022 17:42:05 GMT

try adding .csv to the end <itsi_keys.csv>