Installation

Splunk won't start after upgrading to 8.2.2 from 8.0.x

jfontenot
Explorer

After upgrading Splunk Enterprise to version 8.2.2 from 8.0.x, Splunk will not start on my Indexer/Search head. When I start it I get the following error:

jfontenot_0-1630088453234.png

Any ideas on what could be causing this or places to check?

 

Thanks!

 

Labels (4)
0 Karma

codebuilder
SplunkTrust
SplunkTrust

It looks to me like your filesystem is not mounted. When you created it did you add an entry to fstab?

Check if any files or directories are there: ls -la /mnt
Check fstab: cat /etc/fstab
If it isn't mount your filesystem: mount -a
If it is mounted check that it's not owned by root instead of your Splunk user: Same output from ls -la /mnt
Change user:group ownership to your Splunk user if necessary: chown -RP splunk:splunk /mnt/
(assuming your Splunk user/group is "splunk".

----
An upvote would be appreciated and Accept Solution if it helps!
0 Karma

tscroggins
Motivator

@jfontenot 

Verify all apps that expose web endpoints use Python3. Check each app's web.conf and Python scripts to confirm. Incompatible endpoints will prevent the app server from starting. EDIT: The mobile interface didn't display the other responses when I initially viewed the question. Apologies if you've already checked this.

0 Karma

scelikok
SplunkTrust
SplunkTrust

Please check if exists /mnt/hot or  /mnt mount.  Also, check for permission for splunk user.

If this reply helps you an upvote is appreciated.
0 Karma

s2_splunk
Splunk Employee
Splunk Employee

Have you taken a look at $SPLUNK_HOME/var/log/splunk/splunkd.log to get any hints as to what may be happening? 

0 Karma

jfontenot
Explorer

When I start Splunk I get the following messages in $SPLUNK_HOME/var/log/splunk/splunkd.log.

08-27-2021 13:59:55.720 -0500 WARN DatabaseDirectoryManager [11664 SplunkdSpecificInitThread] - Unable to find a directory for db id=_internal~764~22C4282A-1F3D-4C0D-8517-152DB5BD0C86 with dir_name=hot_v1_764

hot/splunkdb/_internaldb/db' pendingBucketUpdates=1 innerLockTime=0.000. Reason='Getting directory for bid=_internal~764~22C4282A-1F3D-4C0D-8517-152DB5BD0C86 bucket map updated due to missing path="/mnt/hot/splunkdb/_internaldb/db/hot_v1_764"'

08-27-2021 13:59:55.722 -0500 INFO DatabaseDirectoryManager [11664 SplunkdSpecificInitThread] - Finished writing bucket manifest in hotWarmPath=/mnt/hot/splunkdb/_internaldb/db duration=0.001

08-27-2021 13:59:55.723 -0500 WARN DatabaseDirectoryManager [11664 SplunkdSpecificInitThread] - Unable to find a directory for db id=_internal~764~22C4282A-1F3D-4C0D-8517-152DB5BD0C86 with dir_name=hot_v1_764

 

08-27-2021 13:59:55.723 -0500 INFO DatabaseDirectoryManager [11664 SplunkdSpecificInitThread] - idx=_internal writing a bucket manifest in hotWarmPath='/mnt/hot/splunkdb/_internaldb/db' pendingBucketUpdates=1 innerLockTime=0.000. Reason='Getting directory for bid=_internal~764~22C4282A-1F3D-4C0D-8517-152DB5BD0C86 bucket map updated due to missing path="/mnt/hot/splunkdb/_internaldb/db/hot_v1_764"'

08-27-2021 13:59:55.725 -0500 INFO DatabaseDirectoryManager [11664 SplunkdSpecificInitThread] - Finished writing bucket manifest in hotWarmPath=/mnt/hot/splunkdb/_internaldb/db duration=0.002

08-27-2021 13:59:55.725 -0500 WARN DatabaseDirectoryManager [11664 SplunkdSpecificInitThread] - Unable to find a directory for db id=_internal~764~22C4282A-1F3D-4C0D-8517-152DB5BD0C86 with dir_name=hot_v1_764

08-27-2021 13:59:55.726 -0500 INFO DatabaseDirectoryManager [11664 SplunkdSpecificInitThread] - idx=_internal writing a bucket manifest in hotWarmPath='/mnt/hot/splunkdb/_internaldb/db' pendingBucketUpdates=1 innerLockTime=0.000. Reason='Getting directory for bid=_internal~764~22C4282A-1F3D-4C0D-8517-152DB5BD0C86 bucket map updated due to missing path="/mnt/hot/splunkdb/_internaldb/db/hot_v1_764"'

08-27-2021 13:59:55.727 -0500 INFO DatabaseDirectoryManager [11664 SplunkdSpecificInitThread] - Finished writing bucket manifest in hotWarmPath=/mnt/hot/splunkdb/_internaldb/db duration=0.001

08-27-2021 13:59:55.728 -0500 ERROR BucketMover [11664 SplunkdSpecificInitThread] - Unexpected failure to parse bucket='/mnt/cold/splunkdb/_internaldb/colddb/hot_v1_764'

DatabaseDirectoryManager [11664 SplunkdSpecificInitThread] - Unable to find a directory for db id=_internal

0 Karma

sylim_splunk
Splunk Employee
Splunk Employee

The Splunk hot buckets should not end up being in indexname/colddb but it did. You can try to move it back to indexname/db and see if it starts back successfully or move the hotbucket out of the colddb - move it to somewhere like tmp directory - it should come back.

0 Karma

ephemeric
Contributor

You could change in `etc/splunk-launch.conf`:

 

SPLUNK_DB=/tmp/splunk_db

 

 Just to see if you can get a clean start?

I'm probably wrong but it looks like there is a problem binding to port 8000? This could be caused by other errors, like a filesystem problem? I haven't looked at errors like this for years as I am only now making a comeback to the world of Splunk.

0 Karma
Get Updates on the Splunk Community!

Cannot push config from deployer to search head cluster

Hello,I recently upgraded our deployer/deployment server from 8.1.6 to version 9.0 and when I try to push ...

Clear text password in command line

mogod command line argument having clear text password like "--sslPEMKeyPassword=password"how to avoid clear ...

Traffic logs from Splunk Add-on for Cisco Meraki

Recently deployed this add-on, but it doesn't seem to bring back Traffic or URL logs like we did when using ...