The answer is in the question, you encountered a problem with the SHcluster restart order, and it caused the ITSI migration script to not run on the SHcaptain first as expected.
The logic should be :
- trigger rolling restart after pushing the apps from the deployer
- the original shcaptain is the last one to restart
- one it restarts, it triggers the ITSI scripts to create the "global team" object in the kvstore
- then trigger just after the first-install/migration script to setup the rest of ITSI collections in the kvstore
In your case, some SH may have timeout during the rolling restart, or the captain took too long to shutdown and restart.
As a consequence, another shpeer took over and became captain, but as it was already restarted, it did not run the global team creation script.
As you have no team available, the rest of the migration fails for permissions reasons, and the UI is only partially working.
The fix is always the same :
- use the manual script to create the missing global team.
http://docs.splunk.com/Documentation/ITSI/latest/Configure/Installationandconfigurationconsiderationsandissues#Run_script_to_set_the_default_team_to_Global
Run the following command on any search head in your ITSI deployment:
cd $SPLUNK_HOME/etc/apps/SA-ITOA/bin
$SPLUNK_HOME/bin/splunk cmd python itsi_reset_default_team.py
Provide the splunkd port number and Splunk user name and password when prompted.
After the script has successfully finished, the Global group is created in the kvstore.
Restart the Splunk platform.
in the case of Sh-cluster it means a rolling restart of the sh.
As side remark, another reason why the team creation may fail (not just on a SHcluster), is that the role admin is not inheriting from the role itao_admin
see this guide to fix it : http://docs.splunk.com/Documentation/ITSI/latest/Configure/UpgradeSplunkITServiceIntelligence#Check_Splunk_admin_role_inheritance
... View more
