Upgrading from 6.0.3 to 6.1.1 - splunk start hangs with "Checking for possible UI view conflicts".

[andrewkenth]

Just as the title states.

Oddly enough I've upgraded to 6.1.1 from 6.0 3 times w/o issue. This time we start splunk is hanging with the message "Checking for possible UI view conflicts".

Any ideas?

Exact commands used to upgrade.

1) Copy upgrade TAR from DEV to UAT
Log in to u-wcm-splunk
su wcsplnku
mkdir /tmp/splunk_upgrade
scp @d-wcm-splunk:/tmp/splunk_upgrade/* /tmp/splunk_upgrade

2) Copy /apps/wcm-splunk to apps/splunk-bak (or backup anywhere)
mkdir /apps/splunk-bak
rsync -av --progress /apps/wcm-splunk /apps/splunk-bak --exclude archive

3) Stop Splunk
/apps/wcm-splunk/bin/./splunk stop

4) Install upgrade
tar -xvf /tmp/splunk_upgrade/splunk-6.1.1-207789-Linux-x86_64.tar -C /apps/ --transform s/splunk/wcm-splunk/
( chmod -R 755 /apps/wcm-splunk/ if not owned by wcsplnkp users)

5) Copy the follwoing files from the lower environment to the appropriate directories:

cp /tmp/login.css /apps/wcm-splunk/share/splunk/search_mrsparkle/exposed/css/
cp /tmp/web.conf /apps/wcm-splunk/etc/system/local/
cp /tmp/application.css /apps/wcm-splunk/etc/apps/CharlesRiverInfoSec/appserver/static/
cp /tmp/default.css /apps/wcm-splunk/share/splunk/search_mrsparkle/exposed/css/skins/default/

6) Start splunk
/apps/wcm-splunk/bin/./splunk start --accept-license --answer-yes

[smohan_splunk]

Starting from Splunk 6.2.7(SPL-104384) and 6.3.2(SPL-108911) versions, Splunk will limit the scanning to the following subdirectories: $SPLUNK_HOME/etc and $SPLUNK_HOME/share . This way, rest of the directories would not be scanned, and the upgrade procedure to a newer version should be significantly quicker.

[andrewkenth · Accepted solution]

Updating Answers with relevant info.

Upon startup of a new version Splunk ($SPLUNK_HOME/lib/python2.7/site-packages/splunk/clilib/migration.py) scans every file in it's directory structure ($SPLUNK_HOME). Unfortunately for us we have a NAS appliance mounted under $SPLUNK_HOME used for retention purposes that takes a daily snapshot of every file and stores it. 58 files for 58 days. This scan took about 14 hours to complete. The temporary solution is to edit the script (and manage going forward during upgrades until a solution can be found).

Script edit ($SPLUNK_HOME/lib/python2.7/site-packages/splunk/clilib/migration.py), change .snapshot to whatever you'd like to avoid scanning.

def findPreviewFiles():
  # skip over .snapshot subdirectories, which cause sadness when
  # SPLUNK_HOME is a netapp mountpoint with exposed snapshot dirs
  return comm.findFiles(comm.splunk_home, "\\.%s$" % EXT_MIGRATE,
          skipdir_pattern=r"[\\/].snapshot$")

[andrewkenth]

We are running on Redhat Linux.