Splunk ITSI

Shared search for ITSI and glass panel

duffeysplunk
Path Finder

We have recently implemented ITSI into our environment and are building a glass panel which displays the current state of multiple services running on a system. In our case we have about 16-20 services we monitor on each system and about 8 systems. If I used a regular dashboard I could use a base search to power all the panels with 2 or 3 base searches thus reducing load on the system (in terms of # searches). A regular dashboard's layout is too restrictive so we were hoping to do this with glass panels.

I was curious if this was possible in ITSI and how you would implement this?

1 Solution

thejeffreystone
Path Finder

I am not sure what you consider a system, so I just assumed system = host for what I created below.

Just create a base search or searches that gathers your metrics/KPIs by entity. Your entities in this case could be the 16-20 services or it could be the hosts those services are running on. Either way should work as long as long as the base search outputs the metric values by whatever you want to use as your entity.

For example you could setup System1 as a service, and then make your entities the 16-20 services under that service like "system1_service1", "system1_service2" and then your base search gathers your metrics and outputs them in a table of rows where "system1_service1" has some columns of metrics (metric1, metric2 - if each of those service has multiple KPIs) and lookups by entity and filters by entity under service. Then for "system_service1" you can set up some KPIs that use the single base search and pull a single metric for each one of those "systemX_serviceX" entities. On your glass table you could then put all 8 systems as individual services or combine them into a higher service and display the health using those KPIs

Or if those 16-20 services each have just one KPI/metric you could go simpler, and just list your 8 systems as entities, and add those to a service called MyApp. Then create a base search, or multiple that outputs a table with your columns for each of the 16-20 services that contains the metric you want to capture for that service. Then filter the base search by host. Then create a KPI for each of those 16-20 services.

I've just started playing around with ITSI myself, but have already had use cases that used both the approaches above. It really comes down to defining what is a service, entity, and KPI/metric you want to display.

View solution in original post

appache
Path Finder

Hi, YES it is possible to do in ITSI.
when you said you gonna use base search in regular dashboard use the same search in ITSI with out using stats, timechart or chart or table etc and select the aggregation values. which ITSI provides and select the field which you want to have thresholds on in the thresholds field and the you should be good to go. BUT dont forget to have your entities(host, IP etc) in your entities tab.

0 Karma

thejeffreystone
Path Finder

I am not sure what you consider a system, so I just assumed system = host for what I created below.

Just create a base search or searches that gathers your metrics/KPIs by entity. Your entities in this case could be the 16-20 services or it could be the hosts those services are running on. Either way should work as long as long as the base search outputs the metric values by whatever you want to use as your entity.

For example you could setup System1 as a service, and then make your entities the 16-20 services under that service like "system1_service1", "system1_service2" and then your base search gathers your metrics and outputs them in a table of rows where "system1_service1" has some columns of metrics (metric1, metric2 - if each of those service has multiple KPIs) and lookups by entity and filters by entity under service. Then for "system_service1" you can set up some KPIs that use the single base search and pull a single metric for each one of those "systemX_serviceX" entities. On your glass table you could then put all 8 systems as individual services or combine them into a higher service and display the health using those KPIs

Or if those 16-20 services each have just one KPI/metric you could go simpler, and just list your 8 systems as entities, and add those to a service called MyApp. Then create a base search, or multiple that outputs a table with your columns for each of the 16-20 services that contains the metric you want to capture for that service. Then filter the base search by host. Then create a KPI for each of those 16-20 services.

I've just started playing around with ITSI myself, but have already had use cases that used both the approaches above. It really comes down to defining what is a service, entity, and KPI/metric you want to display.

duffeysplunk
Path Finder

Sorry I was not clear about what a system is or what service is.

System: 4 - 8 hosts which are part of a common system
Service: I mean a Windows Service here not the ITSI service (just to be clear)

Based on this I don't think your first solution will work; however, I might be able to define a metric based on the base KPI search. I will try that and let you know the results.

0 Karma

thejeffreystone
Path Finder

Cool.

Yeah, then you just need to know how you want to define your KPIs and how you want those to effect your service.

Sounds like then you could use your 4-8 systems as entities. And then each entity is going to have 16-20+ KPIs that will be used to calculate the health of those entities.

Then your base search could just output a table with the headers like:
host service1 service2 service3

Each service# column would be a metric you could create a KPI around.

0 Karma
Get Updates on the Splunk Community!

Earn a $35 Gift Card for Answering our Splunk Admins & App Developer Survey

Survey for Splunk Admins and App Developers is open now! | Earn a $35 gift card!      Hello there,  Splunk ...

Continuing Innovation & New Integrations Unlock Full Stack Observability For Your ...

You’ve probably heard the latest about AppDynamics joining the Splunk Observability portfolio, deepening our ...

Monitoring Amazon Elastic Kubernetes Service (EKS)

As we’ve seen, integrating Kubernetes environments with Splunk Observability Cloud is a quick and easy way to ...