topic Re: ITSI Error on App Start in Splunk ITSI

ITSI Error on App Start

sloshburch — Wed, 09 Mar 2016 15:58:00 GMT

FYI

Trying to start ITSI for the first time. Even thought these are likely my fault because I've mucked a lot with the environment, I wanted to post them here since I'm sure others are going to run into these at some point and could benefit from the answer. I don't have the answer yet, but I'll post it here when I do. If you already know the answer, please go ahead and claim your karma!

Environment Details

Deployed ITSI in an environment with a Search Head Cluster and Index Cluster.
ITSI version 2.1.0
Splunk Enterprise Instances all on 6.3.3 build f44afce176d0
All AWS EC2 Linux instances

Error Message

A window in the web UI opens showing:

Could not load settings for the page. May need to check user settings for roles and permissions. Details: An internal error has occurred.
Also, getting the following message in the web UI:
Importing IT Service Intelligence settings from conf files ' 'for apps and modules failed with: KV Store is not initialized. We have tried for 2 minutes but KV store still not available

Re: ITSI Error on App Start

sloshburch — Wed, 09 Mar 2016 16:16:40 GMT

Validated that the instructions in this section are implemented: http://docs.splunk.com/Documentation/ITSI/latest/Configure/Setupsearchheadcluster#Map_user_roles_to_LDAP

Potentially correlated error messages:
ERROR KVStoreIntrospection - failed to get introspection data
ERROR KVStorageProvider - An error occurred during the last operation ('serverStatus', domain: '0', code: '0'): No connection available.

Re: ITSI Error on App Start

sloshburch — Wed, 09 Mar 2016 16:45:31 GMT

Just found my that the importRoles from my Burch_zglobal_ta is taking precedence over the same attribute in the itsi app. Thank you btool!

Re: ITSI Error on App Start

sloshburch — Wed, 09 Mar 2016 23:35:01 GMT

Roles look good. Issue persists 😞

Re: ITSI Error on App Start

sloshburch — Wed, 09 Mar 2016 23:40:59 GMT

7756 is my splunkd port. Seeing more errors in the DMC:

Failed to fetch REST endpoint uri=https://127.0.0.1:7756/services/server/introspection/kvstore/collectionstats?count=0 from server=https://127.0.0.1:7756
Unexpected status for to fetch REST endpoint uri=https://127.0.0.1:7756/services/server/introspection/kvstore/collectionstats?count=0 from server=https://127.0.0.1:7756 - Internal Server Error

Re: ITSI Error on App Start

sloshburch — Thu, 10 Mar 2016 00:03:13 GMT

May the mocking and insults begin...

I just noticed that in my lab, which is in AWS, I never enabled a security group inbound rule for my kvstore port. Immediately after fixing that, the KV Store dashboards in the DMC started working...

Rolling restart of SHC to see if that resolved everything (except my sillyness).

Re: ITSI Error on App Start

sloshburch — Thu, 10 Mar 2016 00:14:22 GMT

My hubris is my own weakness.

In my security model of least-privilege access, I never had a reason, and therefore never exposed the kvstore port through my firewall (AWS security group).

So, learn from my knucklehead mistakes. Here's a summary of all of them through this investigation.

Make sure your kvstore port is exposed. Firewall, Security Groups, iptables, etc...
Check your roles. My custom role definitions were taking higher precedence from those role configs needed for ITSI. btool is your friend! http://docs.splunk.com/Documentation/Splunk/latest/Troubleshooting/Usebtooltotroubleshootconfigurations

I think that was it. Good luck to anyone who runs into this same issue. I hope this post helps you out!

Re: ITSI Error on App Start

rajneeshc1981 — Tue, 23 Oct 2018 19:41:11 GMT

hey do you know how to install. the components of ITSI on the free instance.

Re: ITSI Error on App Start

sloshburch — Wed, 24 Oct 2018 12:18:35 GMT

@rajneeshc1981 - I'm a bit confused by the question. Are you asking for the instructions? Did you see the instructions that are in the docs online? It sounds like you're trying to install a premium paid app on a non paid instance of Splunk, so maybe start a new thread on answers and elaborate on what challenge you are trying to solve so we can all help?