ITSI Error on App Start

[sloshburch]

FYI

Trying to start ITSI for the first time. Even thought these are likely my fault because I've mucked a lot with the environment, I wanted to post them here since I'm sure others are going to run into these at some point and could benefit from the answer. I don't have the answer yet, but I'll post it here when I do. If you already know the answer, please go ahead and claim your karma!

Environment Details

• Deployed ITSI in an environment with a Search Head Cluster and Index Cluster.
• ITSI version 2.1.0
• Splunk Enterprise Instances all on 6.3.3 build f44afce176d0
• All AWS EC2 Linux instances

Error Message

A window in the web UI opens showing:

Could not load settings for the page. May need to check user settings for roles and permissions. Details: An internal error has occurred.
Also, getting the following message in the web UI:

Importing IT Service Intelligence settings from conf files ' 'for apps and modules failed with: KV Store is not initialized. We have tried for 2 minutes but KV store still not available

[sloshburch · Accepted solution]

My hubris is my own weakness.

In my security model of least-privilege access, I never had a reason, and therefore never exposed the kvstore port through my firewall (AWS security group).

So, learn from my knucklehead mistakes. Here's a summary of all of them through this investigation.

• Make sure your kvstore port is exposed. Firewall, Security Groups, iptables, etc...
• Check your roles. My custom role definitions were taking higher precedence from those role configs needed for ITSI. btool is your friend! http://docs.splunk.com/Documentation/Splunk/latest/Troubleshooting/Usebtooltotroubleshootconfigurati...

I think that was it. Good luck to anyone who runs into this same issue. I hope this post helps you out!

[rajneeshc1981]

hey do you know how to install. the components of ITSI on the free instance.

[sloshburch]

@rajneeshc1981 - I'm a bit confused by the question. Are you asking for the instructions? Did you see the instructions that are in the docs online? It sounds like you're trying to install a premium paid app on a non paid instance of Splunk, so maybe start a new thread on answers and elaborate on what challenge you are trying to solve so we can all help?

[sloshburch]

May the mocking and insults begin...

I just noticed that in my lab, which is in AWS, I never enabled a security group inbound rule for my kvstore port. Immediately after fixing that, the KV Store dashboards in the DMC started working...

Rolling restart of SHC to see if that resolved everything (except my sillyness).

[sloshburch]

Just found my that the importRoles from my Burch_zglobal_ta is taking precedence over the same attribute in the itsi app. Thank you btool!

[sloshburch]

Roles look good. Issue persists 😞

[sloshburch]

Validated that the instructions in this section are implemented: http://docs.splunk.com/Documentation/ITSI/latest/Configure/Setupsearchheadcluster#Map_user_roles_to_...

Potentially correlated error messages:
ERROR KVStoreIntrospection - failed to get introspection data
ERROR KVStorageProvider - An error occurred during the last operation ('serverStatus', domain: '0', code: '0'): No connection available.

[sloshburch]

7756 is my splunkd port. Seeing more errors in the DMC:

Failed to fetch REST endpoint uri=https://127.0.0.1:7756/services/server/introspection/kvstore/collectionstats?count=0 from server=https://127.0.0.1:7756
Unexpected status for to fetch REST endpoint uri=https://127.0.0.1:7756/services/server/introspection/kvstore/collectionstats?count=0 from server=https://127.0.0.1:7756 - Internal Server Error