Installation

Upgrading from 6.0.3 to 6.1.1 - splunk start hangs with "Checking for possible UI view conflicts".

andrewkenth
Communicator

Just as the title states.

Oddly enough I've upgraded to 6.1.1 from 6.0 3 times w/o issue. This time we start splunk is hanging with the message "Checking for possible UI view conflicts".

Any ideas?

Exact commands used to upgrade.

1) Copy upgrade TAR from DEV to UAT
Log in to u-wcm-splunk
su wcsplnku
mkdir /tmp/splunk_upgrade
scp @d-wcm-splunk:/tmp/splunk_upgrade/* /tmp/splunk_upgrade

2) Copy /apps/wcm-splunk to apps/splunk-bak (or backup anywhere)
mkdir /apps/splunk-bak
rsync -av --progress /apps/wcm-splunk /apps/splunk-bak --exclude archive

3) Stop Splunk
/apps/wcm-splunk/bin/./splunk stop

4) Install upgrade
tar -xvf /tmp/splunk_upgrade/splunk-6.1.1-207789-Linux-x86_64.tar -C /apps/ --transform s/splunk/wcm-splunk/
( chmod -R 755 /apps/wcm-splunk/ if not owned by wcsplnkp users)

5) Copy the follwoing files from the lower environment to the appropriate directories:

cp /tmp/login.css /apps/wcm-splunk/share/splunk/search_mrsparkle/exposed/css/
cp /tmp/web.conf /apps/wcm-splunk/etc/system/local/
cp /tmp/application.css /apps/wcm-splunk/etc/apps/CharlesRiverInfoSec/appserver/static/
cp /tmp/default.css /apps/wcm-splunk/share/splunk/search_mrsparkle/exposed/css/skins/default/

6) Start splunk
/apps/wcm-splunk/bin/./splunk start --accept-license --answer-yes

Labels (1)
Tags (4)
0 Karma
1 Solution

andrewkenth
Communicator

Updating Answers with relevant info.

Upon startup of a new version Splunk ($SPLUNK_HOME/lib/python2.7/site-packages/splunk/clilib/migration.py) scans every file in it's directory structure ($SPLUNK_HOME). Unfortunately for us we have a NAS appliance mounted under $SPLUNK_HOME used for retention purposes that takes a daily snapshot of every file and stores it. 58 files for 58 days. This scan took about 14 hours to complete. The temporary solution is to edit the script (and manage going forward during upgrades until a solution can be found).

Script edit ($SPLUNK_HOME/lib/python2.7/site-packages/splunk/clilib/migration.py), change .snapshot to whatever you'd like to avoid scanning.

def findPreviewFiles():
  # skip over .snapshot subdirectories, which cause sadness when
  # SPLUNK_HOME is a netapp mountpoint with exposed snapshot dirs
  return comm.findFiles(comm.splunk_home, "\\.%s$" % EXT_MIGRATE,
          skipdir_pattern=r"[\\/].snapshot$")

View solution in original post

smohan_splunk
Splunk Employee
Splunk Employee

Starting from Splunk 6.2.7(SPL-104384) and 6.3.2(SPL-108911) versions, Splunk will limit the scanning to the following subdirectories: $SPLUNK_HOME/etc and $SPLUNK_HOME/share . This way, rest of the directories would not be scanned, and the upgrade procedure to a newer version should be significantly quicker.

andrewkenth
Communicator

Updating Answers with relevant info.

Upon startup of a new version Splunk ($SPLUNK_HOME/lib/python2.7/site-packages/splunk/clilib/migration.py) scans every file in it's directory structure ($SPLUNK_HOME). Unfortunately for us we have a NAS appliance mounted under $SPLUNK_HOME used for retention purposes that takes a daily snapshot of every file and stores it. 58 files for 58 days. This scan took about 14 hours to complete. The temporary solution is to edit the script (and manage going forward during upgrades until a solution can be found).

Script edit ($SPLUNK_HOME/lib/python2.7/site-packages/splunk/clilib/migration.py), change .snapshot to whatever you'd like to avoid scanning.

def findPreviewFiles():
  # skip over .snapshot subdirectories, which cause sadness when
  # SPLUNK_HOME is a netapp mountpoint with exposed snapshot dirs
  return comm.findFiles(comm.splunk_home, "\\.%s$" % EXT_MIGRATE,
          skipdir_pattern=r"[\\/].snapshot$")

andrewkenth
Communicator

We are running on Redhat Linux.

0 Karma
Get Updates on the Splunk Community!

.conf24 | Day 0

Hello Splunk Community! My name is Chris, and I'm based in Canberra, Australia's capital, and I travelled for ...

Enhance Security Visibility with Splunk Enterprise Security 7.1 through Threat ...

 (view in My Videos)Struggling with alert fatigue, lack of context, and prioritization around security ...

Troubleshooting the OpenTelemetry Collector

  In this tech talk, you’ll learn how to troubleshoot the OpenTelemetry collector - from checking the ...