Installation

Upgrading from 6.0.3 to 6.1.1 - splunk start hangs with "Checking for possible UI view conflicts".

Communicator

Just as the title states.

Oddly enough I've upgraded to 6.1.1 from 6.0.3 times w/o issue. This time Splunk is hanging with the message "Checking for possible UI view conflicts".

Any ideas?

Exact commands used to upgrade.

1) Copy upgrade TAR from DEV to UAT
Log in to u-wcm-splunk
su wcsplnku
mkdir /tmp/splunkupgrade
scp @d-wcm-splunk:/tmp/splunkupgrade/* /tmp/splunk_upgrade

2) Copy /apps/wcm-splunk to apps/splunk-bak (or backup anywhere)
mkdir /apps/splunk-bak
rsync -av --progress /apps/wcm-splunk /apps/splunk-bak --exclude archive

3) Stop Splunk
/apps/wcm-splunk/bin/./splunk stop

4) Install upgrade
tar -xvf /tmp/splunkupgrade/splunk-6.1.1-207789-Linux-x8664.tar -C /apps/ --transform s/splunk/wcm-splunk/
( chmod -R 755 /apps/wcm-splunk/ if not owned by wcsplnkp users)

5) Copy the follwoing files from the lower environment to the appropriate directories:

cp /tmp/login.css /apps/wcm-splunk/share/splunk/searchmrsparkle/exposed/css/
cp /tmp/web.conf /apps/wcm-splunk/etc/system/local/
cp /tmp/application.css /apps/wcm-splunk/etc/apps/CharlesRiverInfoSec/appserver/static/
cp /tmp/default.css /apps/wcm-splunk/share/splunk/searchmrsparkle/exposed/css/skins/default/

6) Start splunk
/apps/wcm-splunk/bin/./splunk start --accept-license --answer-yes

Labels (1)
Tags (4)
0 Karma
1 Solution

Communicator

Updating Answers with relevant info.

Upon startup of a new version Splunk ($SPLUNKHOME/lib/python2.7/site-packages/splunk/clilib/migration.py) scans every file in it's directory structure ($SPLUNKHOME). Unfortunately for us we have a NAS appliance mounted under $SPLUNK_HOME used for retention purposes that takes a daily snapshot of every file and stores it. 58 files for 58 days. This scan took about 14 hours to complete. The temporary solution is to edit the script (and manage going forward during upgrades until a solution can be found).

Script edit ($SPLUNK_HOME/lib/python2.7/site-packages/splunk/clilib/migration.py), change .snapshot to whatever you'd like to avoid scanning.

def findPreviewFiles():
  # skip over .snapshot subdirectories, which cause sadness when
  # SPLUNK_HOME is a netapp mountpoint with exposed snapshot dirs
  return comm.findFiles(comm.splunk_home, "\\.%s$" % EXT_MIGRATE,
          skipdir_pattern=r"[\\/].snapshot$")

View solution in original post

Splunk Employee
Splunk Employee

Starting from Splunk 6.2.7(SPL-104384) and 6.3.2(SPL-108911) versions, Splunk will limit the scanning to the following subdirectories: $SPLUNKHOME/etc and $SPLUNKHOME/share . This way, rest of the directories would not be scanned, and the upgrade procedure to a newer version should be significantly quicker.

Communicator

Updating Answers with relevant info.

Upon startup of a new version Splunk ($SPLUNKHOME/lib/python2.7/site-packages/splunk/clilib/migration.py) scans every file in it's directory structure ($SPLUNKHOME). Unfortunately for us we have a NAS appliance mounted under $SPLUNK_HOME used for retention purposes that takes a daily snapshot of every file and stores it. 58 files for 58 days. This scan took about 14 hours to complete. The temporary solution is to edit the script (and manage going forward during upgrades until a solution can be found).

Script edit ($SPLUNK_HOME/lib/python2.7/site-packages/splunk/clilib/migration.py), change .snapshot to whatever you'd like to avoid scanning.

def findPreviewFiles():
  # skip over .snapshot subdirectories, which cause sadness when
  # SPLUNK_HOME is a netapp mountpoint with exposed snapshot dirs
  return comm.findFiles(comm.splunk_home, "\\.%s$" % EXT_MIGRATE,
          skipdir_pattern=r"[\\/].snapshot$")

View solution in original post

Communicator

We are running on Redhat Linux.

0 Karma