Splunk ITSI

ITSI Error on App Start

sloshburch
Splunk Employee
Splunk Employee

FYI

Trying to start ITSI for the first time. Even thought these are likely my fault because I've mucked a lot with the environment, I wanted to post them here since I'm sure others are going to run into these at some point and could benefit from the answer. I don't have the answer yet, but I'll post it here when I do. If you already know the answer, please go ahead and claim your karma!

Environment Details

  • Deployed ITSI in an environment with a Search Head Cluster and Index Cluster.
  • ITSI version 2.1.0
  • Splunk Enterprise Instances all on 6.3.3 build f44afce176d0
  • All AWS EC2 Linux instances

Error Message

A window in the web UI opens showing:

Could not load settings for the page. May need to check user settings for roles and permissions. Details: An internal error has occurred.
Also, getting the following message in the web UI:

Importing IT Service Intelligence settings from conf files ' 'for apps and modules failed with: KV Store is not initialized. We have tried for 2 minutes but KV store still not available
Tags (2)
1 Solution

sloshburch
Splunk Employee
Splunk Employee

My hubris is my own weakness.

In my security model of least-privilege access, I never had a reason, and therefore never exposed the kvstore port through my firewall (AWS security group).

So, learn from my knucklehead mistakes. Here's a summary of all of them through this investigation.

I think that was it. Good luck to anyone who runs into this same issue. I hope this post helps you out!

View solution in original post

0 Karma

sloshburch
Splunk Employee
Splunk Employee

My hubris is my own weakness.

In my security model of least-privilege access, I never had a reason, and therefore never exposed the kvstore port through my firewall (AWS security group).

So, learn from my knucklehead mistakes. Here's a summary of all of them through this investigation.

I think that was it. Good luck to anyone who runs into this same issue. I hope this post helps you out!

0 Karma

rajneeshc1981
Explorer

hey do you know how to install. the components of ITSI on the free instance.

0 Karma

sloshburch
Splunk Employee
Splunk Employee

@rajneeshc1981 - I'm a bit confused by the question. Are you asking for the instructions? Did you see the instructions that are in the docs online? It sounds like you're trying to install a premium paid app on a non paid instance of Splunk, so maybe start a new thread on answers and elaborate on what challenge you are trying to solve so we can all help?

0 Karma

sloshburch
Splunk Employee
Splunk Employee

May the mocking and insults begin...

I just noticed that in my lab, which is in AWS, I never enabled a security group inbound rule for my kvstore port. Immediately after fixing that, the KV Store dashboards in the DMC started working...

Rolling restart of SHC to see if that resolved everything (except my sillyness).

0 Karma

sloshburch
Splunk Employee
Splunk Employee

Just found my that the importRoles from my Burch_zglobal_ta is taking precedence over the same attribute in the itsi app. Thank you btool!

0 Karma

sloshburch
Splunk Employee
Splunk Employee

Roles look good. Issue persists 😞

0 Karma

sloshburch
Splunk Employee
Splunk Employee

Validated that the instructions in this section are implemented: http://docs.splunk.com/Documentation/ITSI/latest/Configure/Setupsearchheadcluster#Map_user_roles_to_...

Potentially correlated error messages:
ERROR KVStoreIntrospection - failed to get introspection data
ERROR KVStorageProvider - An error occurred during the last operation ('serverStatus', domain: '0', code: '0'): No connection available.

0 Karma

sloshburch
Splunk Employee
Splunk Employee

7756 is my splunkd port. Seeing more errors in the DMC:

Failed to fetch REST endpoint uri=https://127.0.0.1:7756/services/server/introspection/kvstore/collectionstats?count=0 from server=https://127.0.0.1:7756
Unexpected status for to fetch REST endpoint uri=https://127.0.0.1:7756/services/server/introspection/kvstore/collectionstats?count=0 from server=https://127.0.0.1:7756 - Internal Server Error

0 Karma
Get Updates on the Splunk Community!

New in Observability - Improvements to Custom Metrics SLOs, Log Observer Connect & ...

The latest enhancements to the Splunk observability portfolio deliver improved SLO management accuracy, better ...

Improve Data Pipelines Using Splunk Data Management

  Register Now   This Tech Talk will explore the pipeline management offerings Edge Processor and Ingest ...

3-2-1 Go! How Fast Can You Debug Microservices with Observability Cloud?

Register Join this Tech Talk to learn how unique features like Service Centric Views, Tag Spotlight, and ...