Splunk won't start after upgrading to 8.2.2 from 8.0.x

[jfontenot]

After upgrading Splunk Enterprise to version 8.2.2 from 8.0.x, Splunk will not start on my Indexer/Search head. When I start it I get the following error:

Any ideas on what could be causing this or places to check?

Thanks!

[codebuilder]

It looks to me like your filesystem is not mounted. When you created it did you add an entry to fstab?

Check if any files or directories are there: ls -la /mnt
Check fstab: cat /etc/fstab
If it isn't mount your filesystem: mount -a
If it is mounted check that it's not owned by root instead of your Splunk user: Same output from ls -la /mnt
Change user:group ownership to your Splunk user if necessary: chown -RP splunk:splunk /mnt/
(assuming your Splunk user/group is "splunk".

[tscroggins]

@jfontenot 

Verify all apps that expose web endpoints use Python3. Check each app's web.conf and Python scripts to confirm. Incompatible endpoints will prevent the app server from starting. EDIT: The mobile interface didn't display the other responses when I initially viewed the question. Apologies if you've already checked this.

[scelikok]

Please check if exists /mnt/hot or  /mnt mount.  Also, check for permission for splunk user.

[s2_splunk]

Have you taken a look at $SPLUNK_HOME/var/log/splunk/splunkd.log to get any hints as to what may be happening? 

[jfontenot]

When I start Splunk I get the following messages in $SPLUNK_HOME/var/log/splunk/splunkd.log.

08-27-2021 13:59:55.720 -0500 WARN DatabaseDirectoryManager [11664 SplunkdSpecificInitThread] - Unable to find a directory for db id=_internal~764~22C4282A-1F3D-4C0D-8517-152DB5BD0C86 with dir_name=hot_v1_764

hot/splunkdb/_internaldb/db' pendingBucketUpdates=1 innerLockTime=0.000. Reason='Getting directory for bid=_internal~764~22C4282A-1F3D-4C0D-8517-152DB5BD0C86 bucket map updated due to missing path="/mnt/hot/splunkdb/_internaldb/db/hot_v1_764"'

08-27-2021 13:59:55.722 -0500 INFO DatabaseDirectoryManager [11664 SplunkdSpecificInitThread] - Finished writing bucket manifest in hotWarmPath=/mnt/hot/splunkdb/_internaldb/db duration=0.001

08-27-2021 13:59:55.723 -0500 WARN DatabaseDirectoryManager [11664 SplunkdSpecificInitThread] - Unable to find a directory for db id=_internal~764~22C4282A-1F3D-4C0D-8517-152DB5BD0C86 with dir_name=hot_v1_764

08-27-2021 13:59:55.723 -0500 INFO DatabaseDirectoryManager [11664 SplunkdSpecificInitThread] - idx=_internal writing a bucket manifest in hotWarmPath='/mnt/hot/splunkdb/_internaldb/db' pendingBucketUpdates=1 innerLockTime=0.000. Reason='Getting directory for bid=_internal~764~22C4282A-1F3D-4C0D-8517-152DB5BD0C86 bucket map updated due to missing path="/mnt/hot/splunkdb/_internaldb/db/hot_v1_764"'

08-27-2021 13:59:55.725 -0500 INFO DatabaseDirectoryManager [11664 SplunkdSpecificInitThread] - Finished writing bucket manifest in hotWarmPath=/mnt/hot/splunkdb/_internaldb/db duration=0.002

08-27-2021 13:59:55.725 -0500 WARN DatabaseDirectoryManager [11664 SplunkdSpecificInitThread] - Unable to find a directory for db id=_internal~764~22C4282A-1F3D-4C0D-8517-152DB5BD0C86 with dir_name=hot_v1_764

08-27-2021 13:59:55.726 -0500 INFO DatabaseDirectoryManager [11664 SplunkdSpecificInitThread] - idx=_internal writing a bucket manifest in hotWarmPath='/mnt/hot/splunkdb/_internaldb/db' pendingBucketUpdates=1 innerLockTime=0.000. Reason='Getting directory for bid=_internal~764~22C4282A-1F3D-4C0D-8517-152DB5BD0C86 bucket map updated due to missing path="/mnt/hot/splunkdb/_internaldb/db/hot_v1_764"'

08-27-2021 13:59:55.727 -0500 INFO DatabaseDirectoryManager [11664 SplunkdSpecificInitThread] - Finished writing bucket manifest in hotWarmPath=/mnt/hot/splunkdb/_internaldb/db duration=0.001

08-27-2021 13:59:55.728 -0500 ERROR BucketMover [11664 SplunkdSpecificInitThread] - Unexpected failure to parse bucket='/mnt/cold/splunkdb/_internaldb/colddb/hot_v1_764'

DatabaseDirectoryManager [11664 SplunkdSpecificInitThread] - Unable to find a directory for db id=_internal

[sylim_splunk]

The Splunk hot buckets should not end up being in indexname/colddb but it did. You can try to move it back to indexname/db and see if it starts back successfully or move the hotbucket out of the colddb - move it to somewhere like tmp directory - it should come back.

[ephemeric]

You could change in `etc/splunk-launch.conf`:

SPLUNK_DB=/tmp/splunk_db

 Just to see if you can get a clean start?

I'm probably wrong but it looks like there is a problem binding to port 8000? This could be caused by other errors, like a filesystem problem? I haven't looked at errors like this for years as I am only now making a comeback to the world of Splunk.