Why is the fresh install of ITSI 3.1.2 onto a search head cluster environment not working?

[sylim_splunk]

Here's the steps I took for installation and what I get soon after clicking on ITSI app.
Steps:

1. Untar splunk-it-service-intelligence_312.spl into etc/shcluster/apps
2. push new bundle down to search head cluster members
3. Log in to splunkWeb and click "IT Service Intelligence" app.
4. Error message pops up on the screen and clicking "OK" button leads me to the screen that appear to be stuck at "loading". Installation was super easy .. just took a few minutes. But why am I getting the 403 error? Could not load settings for the page. Status: 403 (Forbiddne) Details: "admin" does not have the capability "write_itsi_homeview"

[spkriyaz]

@sylim_splunk currently i am facing role related issue in ITSI. I installed ITSI 4.3.1 version in Splunk enterprise 8.0.3 and after successful installation when I open the ITSI app the below error pops out saying "Could not load page settings. Check that you have the proper roles and permissions. Details: Page not found!" and when I try to open other options like glass tables, deep dive etc. it throws another error saying

"Deep Dive could not be loaded. Possible cause: connection lost. Try restarting the Splunk platform. Status: 404 (Not Found) Details: Page not found!"

Below is my authorise list for role_admin which looks ok but not sure why the above errors occur. Could you please help with your expertise. I have attached the screenshot as well.

C:\Program Files\Splunk\bin>splunk btool authorize list role_admin
[role_admin]
accelerate_datamodel = enabled
admin_all_objects = enabled
apps_backup = enabled
apps_restore = enabled
change_authentication = enabled
cumulativeRTSrchJobsQuota = 400
cumulativeSrchJobsQuota = 200
dispatch_rest_to_indexers = disabled
edit_authentication_extensions = enabled
edit_bookmarks_mc = enabled
edit_cmd = enabled
edit_deployment_client = enabled
edit_deployment_server = enabled
edit_dist_peer = enabled
edit_encryption_key_provider = enabled
edit_forwarders = enabled
edit_health = enabled
edit_httpauths = enabled
edit_indexer_cluster = enabled
edit_indexerdiscovery = enabled
edit_input_defaults = enabled
edit_local_apps = enabled
edit_metric_schema = enabled
edit_metrics_rollup = enabled
edit_modinput_admon = enabled
edit_modinput_perfmon = enabled
edit_modinput_winhostmon = enabled
edit_modinput_winnetmon = enabled
edit_modinput_winprintmon = enabled
edit_monitor = enabled
edit_restmap = enabled
edit_roles = enabled
edit_scripted = enabled
edit_search_concurrency_all = enabled
edit_search_head_clustering = enabled
edit_search_schedule_priority = enabled
edit_search_scheduler = enabled
edit_search_server = enabled
edit_server = enabled
edit_server_crl = enabled
edit_splunktcp = enabled
edit_splunktcp_ssl = enabled
edit_splunktcp_token = enabled
edit_tcp = enabled
edit_tcp_stream = enabled
edit_telemetry_settings = enabled
edit_token_http = disabled
edit_tokens_all = enabled
edit_tokens_own = enabled
edit_tokens_settings = enabled
edit_udp = enabled
edit_upload_and_index = enabled
edit_user = enabled
edit_view_html = enabled
edit_web_settings = enabled
edit_win_eventlogs = enabled
edit_win_regmon = enabled
edit_win_wmiconf = enabled
edit_workload_pools = enabled
edit_workload_rules = enabled
get_diag = enabled
grantableRoles = admin
importRoles = itoa_admin;itoa_analyst;itoa_user;power;user
indexes_edit = enabled
install_apps = enabled
license_edit = enabled
license_tab = enabled
license_view_warnings = enabled
list_cascading_plans = enabled
list_deployment_client = enabled
list_deployment_server = enabled
list_dist_peer = enabled
list_forwarders = enabled
list_health = enabled
list_httpauths = enabled
list_indexer_cluster = enabled
list_indexerdiscovery = enabled
list_pdfserver = enabled
list_pipeline_sets = enabled
list_search_head_clustering = disabled
list_search_scheduler = enabled
list_settings = disabled
list_storage_passwords = disabled
list_tokens_all = enabled
list_win_localavailablelogs = enabled
list_workload_pools = enabled
list_workload_rules = enabled
never_expire = enabled
never_lockout = enabled
read_metric_ad = disabled
refresh_application_licenses = enabled
rest_apps_management = enabled
restart_reason = enabled
restart_splunkd = enabled
rtSrchJobsQuota = 100
run_collect = enabled
run_debug_commands = enabled
run_mcollect = enabled
run_msearch = enabled
schedule_rtsearch = enabled
select_workload_pools = enabled
srchDiskQuota = 25000
srchFilter = *
srchFilterSelecting = true
srchIndexesAllowed = *;_*;itsi_grouped_alerts;itsi_notable_archive;itsi_notable_audit;itsi_summary;itsi_tracked_alerts
srchIndexesDefault = main
srchJobsQuota = 50
srchMaxTime = 8640000
srchTimeWin = 0
web_debug = enabled
write_metric_ad = disabled
write_pdfserver = enabled

[sylim_splunk · Accepted solution]

It turned out to be hitting an unexpected corner case. SHCluster used to have ES installed and now moved out of it - but it had side-effect of causing this error.
As the error message suggests it appears something to do with capability. Firstly checked the role_admin which is supposed to inherit itoa_admin via GUI but it didn't have or use "splunk btool --debug authorize list role_admin ".

By default it is included in "apps/itsi/default/authorize.conf" but it's been overwritten by a left-over from ES install setup for the role.

[role_admin] in etc/system/local/authorize.conf
importRoles = can_delete;ess_admin;ess_analyst;ess_user;power;user

After adding itoa_admin to role_admin it immediately allows me to get in.
There is a documentation addressing the similar situation but unfortuately it is under "Upgrade Splunk ITSI" section.
http://docs.splunk.com/Documentation/ITSI/3.1.2/Configure/UpgradeSplunkITServiceIntelligence

It would have been better if it is under steps for installation page. Just FYI,

Before upgrading, make sure the Splunk admin role inherits from the itoa_admin role. The default settings for admin role inheritance for ITSI are contained in $SPLUNK_HOME/etc/apps/itsi/default/authorize.conf. Problems can occur when these settings have been modified in $SPLUNK_HOME/etc/system/local/authorize.conf which takes precedence over the ITSI .conf file settings.

Do the following:

Use the CLI btool command and look at the line importRoles to make sure itoa_admin, itoa_analyst, and itoa_user are listed. For example:

./splunk btool authorize list role_admin

To add the itoa roles, do one of the following: From the UI, navigate to Settings > Access Controls> Roles > admin > Inheritance. Add itoa_admin, itoa_analyst and itoa_user to Selected roles if necessary. Alternatively, open $SPLUNK_HOME/etc/system/local/authorize.conf. Make sure itoa_admin, itoa_analyst and itoa_user are listed in the [role_admin] stanza for the importRoles setting as shown below.

[role_admin]
importRoles = itoa_admin;itoa_analyst;itoa_user;power;user

If they are not, add them manually.

Even if it still doesn't allow you to get in, check this out and try "splunk cmd python itsi_reset_default_team.py"
http://docs.splunk.com/Documentation/ITSI/3.0.1/Configure/Installationandconfigurationconsiderations...