Getting Data In

Why am I getting "homePath='/opt/splunk/var/lib/splunk/audit/db' of index=_audit on unusable filesystem." while starting Splunk on an indexer?

ayushchoudhary
Path Finder

I got this error while starting Splunk on the indexer.

homePath='/opt/splunk/var/lib/splunk/audit/db' of index=_audit on unusable filesystem. Validating databases (splunkd validatedb) failed with code '1'. 

Please help urgently.

1 Solution

naisanza
Path Finder

You'll want to append the following configuration option to $SPLUNK_HOME/etc/splunk-launch.conf:

OPTIMISTIC_ABOUT_FILE_LOCKING = 1

Note from malmoore (Splunk): As of 28 March 2018, this workaround has been officially documented in the Troubleshooting Manual. See Splunk Enterprise does not start due to unusable filesystem in the manual for the procedure.

The caveats for using this workaround still apply. Proceed with caution, and at your own risk. Irrevocable data loss can still occur. We have already had one report in this thread of problems that have occurred after enabling this setting.

Key points (as of 24 April 2018)

  • There is still no support for macOS 10.13 High Sierra on Splunk Enterprise version 7.0.
  • There is work scheduled to fix the problem for macOS 10.13 on Splunk Enterprise 7.0 and reinstate support, but there is no promise of delivery of this functionality.
  • There is support for macOS 10.13 High Sierra on APFS on Splunk Enterprise version 7.1.

View solution in original post

jmantor
Path Finder

I just encountered this same error running Splunk 6.5.6 on RHEL with an EXT4 file system.

0 Karma

malmoore
Splunk Employee
Splunk Employee

Is this a fresh install?
What version of RHEL?
Did you upgrade and switch from another file system to ext4?

0 Karma

jmantor
Path Finder

It was an existing install. RHEL 6.x. It turns out the lun that the disk was on was accidentally filled up via a VMware snapshot.

0 Karma

malmoore
Splunk Employee
Splunk Employee

Ah, thanks or the info. Another reason why setting this variable should be done only as a last resort.

0 Karma

bnariyani_splun
Splunk Employee
Splunk Employee

Worked Well...

0 Karma

rcrohns_splunk
Splunk Employee
Splunk Employee

This worked for me on macOS High Sierra 10.13.3 with Splunk version 7.0.2.,

uthornander_spl
Splunk Employee
Splunk Employee

Is this still applicable to 7.1?

UT
0 Karma

sloshburch
Splunk Employee
Splunk Employee

Did you mean Splunk Enterprise 7.0.1 or an OS version?

0 Karma

uthornander_spl
Splunk Employee
Splunk Employee

Is this still applicable to 7.1?

UT
0 Karma

tjr1775
Path Finder

So I had this problem as well, and the optimistic file thingee =1 did not work. I have MacOS 10.13.1 (High Sierra) and Splunk 7.0. It was thought this wasn't a problem with 7.0, but it is. However, here is the fix if the optimistic thing is well, not so optimistic:

rm /opt/splunk/lib/libz.1.dylib
cp /usr/lib/libz.1.dylib /opt/splunk/lib/libz.1.dylib

Found at a similar thread: https://answers.splunk.com/answers/585512/importerror-symbol-not-found-inflatevalidate-when.html

pranavna
Explorer

that fixed my issue. thanks...

0 Karma

s2_splunk
Splunk Employee
Splunk Employee

Careful. This is an indication that you may have Splunk deployed on top of an unsupported filesystem that does not implement required file locking mechanism. Setting that attribute in splunk-launch.conf is overriding our internal file locking test during startup. YMMV...

vbumgarner
Contributor

After using this flag for awhile, I'm now getting:

WARN JournalSlice - Error reading from fresh journal slice file ".../db/hot_v1_4937/rawdata/1971039751": Input/output error

Is this related, or do I just have a bad disk?

0 Karma

Dimitri_McKay
Splunk Employee
Splunk Employee

Actually, this worked perfect for me too. I'm running Sierra beta and two of my instances complained about this. So it may be OS X Beta related.

0 Karma

vbumgarner
Contributor

This is still required on the public release of Sierra.

0 Karma

naisanza
Path Finder

You'll want to append the following configuration option to $SPLUNK_HOME/etc/splunk-launch.conf:

OPTIMISTIC_ABOUT_FILE_LOCKING = 1

Note from malmoore (Splunk): As of 28 March 2018, this workaround has been officially documented in the Troubleshooting Manual. See Splunk Enterprise does not start due to unusable filesystem in the manual for the procedure.

The caveats for using this workaround still apply. Proceed with caution, and at your own risk. Irrevocable data loss can still occur. We have already had one report in this thread of problems that have occurred after enabling this setting.

Key points (as of 24 April 2018)

  • There is still no support for macOS 10.13 High Sierra on Splunk Enterprise version 7.0.
  • There is work scheduled to fix the problem for macOS 10.13 on Splunk Enterprise 7.0 and reinstate support, but there is no promise of delivery of this functionality.
  • There is support for macOS 10.13 High Sierra on APFS on Splunk Enterprise version 7.1.

im_bharath
Path Finder

Hey @naisanza

I installed the splunk on the Ubuntu (WSL) and encountered the same issue and i have tried the option you have provided and it worked.. Thank you very much. 

0 Karma

ipapa_splunk
Splunk Employee
Splunk Employee

I had the same issue on MacOS High Sierra after the upgrade from Sierra. You just need to had this line of code and it works. Tested with Splunk Enterprise 7.0

0 Karma

nitinkul
New Member

This worked for me..thank you.

0 Karma

Jason_S
Path Finder
0 Karma
Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

Splunk is officially part of Cisco

Revolutionizing how our customers build resilience across their entire digital footprint.   Splunk ...

Splunk APM & RUM | Planned Maintenance March 26 - March 28, 2024

There will be planned maintenance for Splunk APM and RUM between March 26, 2024 and March 28, 2024 as ...