Why am I getting "homePath='/opt/splunk/var/lib/splunk/audit/db' of index=_audit on unusable filesystem." while starting Splunk on an indexer?

[ayushchoudhary]

I got this error while starting Splunk on the indexer.

homePath='/opt/splunk/var/lib/splunk/audit/db' of index=_audit on unusable filesystem. Validating databases (splunkd validatedb) failed with code '1'. 

Please help urgently.

[jmantor]

I just encountered this same error running Splunk 6.5.6 on RHEL with an EXT4 file system.

[malmoore]

Is this a fresh install?
What version of RHEL?
Did you upgrade and switch from another file system to ext4?

[jmantor]

It was an existing install. RHEL 6.x. It turns out the lun that the disk was on was accidentally filled up via a VMware snapshot.

[malmoore]

Ah, thanks or the info. Another reason why setting this variable should be done only as a last resort.

[bnariyani_splun]

Worked Well...

[rcrohns_splunk]

This worked for me on macOS High Sierra 10.13.3 with Splunk version 7.0.2.,

[uthornander_spl]

Is this still applicable to 7.1?

[sloshburch]

Did you mean Splunk Enterprise 7.0.1 or an OS version?

[uthornander_spl]

Is this still applicable to 7.1?

[tjr1775]

So I had this problem as well, and the optimistic file thingee =1 did not work. I have MacOS 10.13.1 (High Sierra) and Splunk 7.0. It was thought this wasn't a problem with 7.0, but it is. However, here is the fix if the optimistic thing is well, not so optimistic:

rm /opt/splunk/lib/libz.1.dylib
cp /usr/lib/libz.1.dylib /opt/splunk/lib/libz.1.dylib

Found at a similar thread: https://answers.splunk.com/answers/585512/importerror-symbol-not-found-inflatevalidate-when.html

[pranavna]

that fixed my issue. thanks...

[s2_splunk]

Careful. This is an indication that you may have Splunk deployed on top of an unsupported filesystem that does not implement required file locking mechanism. Setting that attribute in splunk-launch.conf is overriding our internal file locking test during startup. YMMV...

[vbumgarner]

After using this flag for awhile, I'm now getting:

WARN JournalSlice - Error reading from fresh journal slice file ".../db/hot_v1_4937/rawdata/1971039751": Input/output error

Is this related, or do I just have a bad disk?

[Dimitri_McKay]

Actually, this worked perfect for me too. I'm running Sierra beta and two of my instances complained about this. So it may be OS X Beta related.

[vbumgarner]

This is still required on the public release of Sierra.

[naisanza · Accepted solution]

You'll want to append the following configuration option to $SPLUNK_HOME/etc/splunk-launch.conf:

OPTIMISTIC_ABOUT_FILE_LOCKING = 1

Note from malmoore (Splunk): As of 28 March 2018, this workaround has been officially documented in the Troubleshooting Manual. See Splunk Enterprise does not start due to unusable filesystem in the manual for the procedure.

The caveats for using this workaround still apply. Proceed with caution, and at your own risk. Irrevocable data loss can still occur. We have already had one report in this thread of problems that have occurred after enabling this setting.

Key points (as of 24 April 2018)

• There is still no support for macOS 10.13 High Sierra on Splunk Enterprise version 7.0.
• There is work scheduled to fix the problem for macOS 10.13 on Splunk Enterprise 7.0 and reinstate support, but there is no promise of delivery of this functionality.
• There is support for macOS 10.13 High Sierra on APFS on Splunk Enterprise version 7.1.

[im_bharath]

Hey @naisanza, 

I installed the splunk on the Ubuntu (WSL) and encountered the same issue and i have tried the option you have provided and it worked.. Thank you very much. 

[ipapa_splunk]

I had the same issue on MacOS High Sierra after the upgrade from Sierra. You just need to had this line of code and it works. Tested with Splunk Enterprise 7.0

[nitinkul]

This worked for me..thank you.

[Jason_S]

Splunk 7.1.0 now supports High Sierra (10.13):

http://docs.splunk.com/Documentation/Splunk/7.1.0/Installation/Systemrequirements#Unix_operating_sys...