Installation

Upgrading from 6.0.3 to 6.1.1 - splunk start hangs with "Checking for possible UI view conflicts".

andrewkenth
Communicator

Just as the title states.

Oddly enough I've upgraded to 6.1.1 from 6.0 3 times w/o issue. This time we start splunk is hanging with the message "Checking for possible UI view conflicts".

Any ideas?

Exact commands used to upgrade.

1) Copy upgrade TAR from DEV to UAT
Log in to u-wcm-splunk
su wcsplnku
mkdir /tmp/splunk_upgrade
scp @d-wcm-splunk:/tmp/splunk_upgrade/* /tmp/splunk_upgrade

2) Copy /apps/wcm-splunk to apps/splunk-bak (or backup anywhere)
mkdir /apps/splunk-bak
rsync -av --progress /apps/wcm-splunk /apps/splunk-bak --exclude archive

3) Stop Splunk
/apps/wcm-splunk/bin/./splunk stop

4) Install upgrade
tar -xvf /tmp/splunk_upgrade/splunk-6.1.1-207789-Linux-x86_64.tar -C /apps/ --transform s/splunk/wcm-splunk/
( chmod -R 755 /apps/wcm-splunk/ if not owned by wcsplnkp users)

5) Copy the follwoing files from the lower environment to the appropriate directories:

cp /tmp/login.css /apps/wcm-splunk/share/splunk/search_mrsparkle/exposed/css/
cp /tmp/web.conf /apps/wcm-splunk/etc/system/local/
cp /tmp/application.css /apps/wcm-splunk/etc/apps/CharlesRiverInfoSec/appserver/static/
cp /tmp/default.css /apps/wcm-splunk/share/splunk/search_mrsparkle/exposed/css/skins/default/

6) Start splunk
/apps/wcm-splunk/bin/./splunk start --accept-license --answer-yes

Labels (1)
Tags (4)
0 Karma
1 Solution

andrewkenth
Communicator

Updating Answers with relevant info.

Upon startup of a new version Splunk ($SPLUNK_HOME/lib/python2.7/site-packages/splunk/clilib/migration.py) scans every file in it's directory structure ($SPLUNK_HOME). Unfortunately for us we have a NAS appliance mounted under $SPLUNK_HOME used for retention purposes that takes a daily snapshot of every file and stores it. 58 files for 58 days. This scan took about 14 hours to complete. The temporary solution is to edit the script (and manage going forward during upgrades until a solution can be found).

Script edit ($SPLUNK_HOME/lib/python2.7/site-packages/splunk/clilib/migration.py), change .snapshot to whatever you'd like to avoid scanning.

def findPreviewFiles():
  # skip over .snapshot subdirectories, which cause sadness when
  # SPLUNK_HOME is a netapp mountpoint with exposed snapshot dirs
  return comm.findFiles(comm.splunk_home, "\\.%s$" % EXT_MIGRATE,
          skipdir_pattern=r"[\\/].snapshot$")

View solution in original post

smohan_splunk
Splunk Employee
Splunk Employee

Starting from Splunk 6.2.7(SPL-104384) and 6.3.2(SPL-108911) versions, Splunk will limit the scanning to the following subdirectories: $SPLUNK_HOME/etc and $SPLUNK_HOME/share . This way, rest of the directories would not be scanned, and the upgrade procedure to a newer version should be significantly quicker.

andrewkenth
Communicator

Updating Answers with relevant info.

Upon startup of a new version Splunk ($SPLUNK_HOME/lib/python2.7/site-packages/splunk/clilib/migration.py) scans every file in it's directory structure ($SPLUNK_HOME). Unfortunately for us we have a NAS appliance mounted under $SPLUNK_HOME used for retention purposes that takes a daily snapshot of every file and stores it. 58 files for 58 days. This scan took about 14 hours to complete. The temporary solution is to edit the script (and manage going forward during upgrades until a solution can be found).

Script edit ($SPLUNK_HOME/lib/python2.7/site-packages/splunk/clilib/migration.py), change .snapshot to whatever you'd like to avoid scanning.

def findPreviewFiles():
  # skip over .snapshot subdirectories, which cause sadness when
  # SPLUNK_HOME is a netapp mountpoint with exposed snapshot dirs
  return comm.findFiles(comm.splunk_home, "\\.%s$" % EXT_MIGRATE,
          skipdir_pattern=r"[\\/].snapshot$")

andrewkenth
Communicator

We are running on Redhat Linux.

0 Karma
Get Updates on the Splunk Community!

Maximize the Value from Microsoft Defender with Splunk

<P style=" text-align: center; "><span class="lia-inline-image-display-wrapper lia-image-align-center" ...

This Week's Community Digest - Splunk Community Happenings [6.27.22]

<FONT size="5"><FONT size="5" color="#FF00FF">Get the latest news and updates from the Splunk Community ...