Getting Data In

Why am I getting "homePath='/opt/splunk/var/lib/splunk/audit/db' of index=_audit on unusable filesystem." while starting Splunk on an indexer?

Path Finder

I got this error while starting Splunk on the indexer.

homePath='/opt/splunk/var/lib/splunk/audit/db' of index=_audit on unusable filesystem. Validating databases (splunkd validatedb) failed with code '1'. 

Please help urgently.

1 Solution

Path Finder

You'll want to append the following configuration option to $SPLUNK_HOME/etc/splunk-launch.conf:

OPTIMISTIC_ABOUT_FILE_LOCKING = 1

Note from malmoore (Splunk): As of 28 March 2018, this workaround has been officially documented in the Troubleshooting Manual. See Splunk Enterprise does not start due to unusable filesystem in the manual for the procedure.

The caveats for using this workaround still apply. Proceed with caution, and at your own risk. Irrevocable data loss can still occur. We have already had one report in this thread of problems that have occurred after enabling this setting.

Key points (as of 24 April 2018)

  • There is still no support for macOS 10.13 High Sierra on Splunk Enterprise version 7.0.
  • There is work scheduled to fix the problem for macOS 10.13 on Splunk Enterprise 7.0 and reinstate support, but there is no promise of delivery of this functionality.
  • There is support for macOS 10.13 High Sierra on APFS on Splunk Enterprise version 7.1.

View solution in original post

Path Finder

I just encountered this same error running Splunk 6.5.6 on RHEL with an EXT4 file system.

0 Karma

Splunk Employee
Splunk Employee

Is this a fresh install?
What version of RHEL?
Did you upgrade and switch from another file system to ext4?

0 Karma

Path Finder

It was an existing install. RHEL 6.x. It turns out the lun that the disk was on was accidentally filled up via a VMware snapshot.

0 Karma

Splunk Employee
Splunk Employee

Ah, thanks or the info. Another reason why setting this variable should be done only as a last resort.

0 Karma

Splunk Employee
Splunk Employee

Worked Well...

0 Karma

Splunk Employee
Splunk Employee

This worked for me on macOS High Sierra 10.13.3 with Splunk version 7.0.2.,

Splunk Employee
Splunk Employee

Is this still applicable to 7.1?

0 Karma

Ultra Champion

Did you mean Splunk Enterprise 7.0.1 or an OS version?

0 Karma

Splunk Employee
Splunk Employee

Is this still applicable to 7.1?

0 Karma

Path Finder

So I had this problem as well, and the optimistic file thingee =1 did not work. I have MacOS 10.13.1 (High Sierra) and Splunk 7.0. It was thought this wasn't a problem with 7.0, but it is. However, here is the fix if the optimistic thing is well, not so optimistic:

rm /opt/splunk/lib/libz.1.dylib
cp /usr/lib/libz.1.dylib /opt/splunk/lib/libz.1.dylib

Found at a similar thread: https://answers.splunk.com/answers/585512/importerror-symbol-not-found-inflatevalidate-when.html

Explorer

that fixed my issue. thanks...

0 Karma

Splunk Employee
Splunk Employee

Careful. This is an indication that you may have Splunk deployed on top of an unsupported filesystem that does not implement required file locking mechanism. Setting that attribute in splunk-launch.conf is overriding our internal file locking test during startup. YMMV...

Contributor

After using this flag for awhile, I'm now getting:

WARN JournalSlice - Error reading from fresh journal slice file ".../db/hot_v1_4937/rawdata/1971039751": Input/output error

Is this related, or do I just have a bad disk?

0 Karma

Splunk Employee
Splunk Employee

Actually, this worked perfect for me too. I'm running Sierra beta and two of my instances complained about this. So it may be OS X Beta related.

0 Karma

Contributor

This is still required on the public release of Sierra.

0 Karma

Path Finder

You'll want to append the following configuration option to $SPLUNK_HOME/etc/splunk-launch.conf:

OPTIMISTIC_ABOUT_FILE_LOCKING = 1

Note from malmoore (Splunk): As of 28 March 2018, this workaround has been officially documented in the Troubleshooting Manual. See Splunk Enterprise does not start due to unusable filesystem in the manual for the procedure.

The caveats for using this workaround still apply. Proceed with caution, and at your own risk. Irrevocable data loss can still occur. We have already had one report in this thread of problems that have occurred after enabling this setting.

Key points (as of 24 April 2018)

  • There is still no support for macOS 10.13 High Sierra on Splunk Enterprise version 7.0.
  • There is work scheduled to fix the problem for macOS 10.13 on Splunk Enterprise 7.0 and reinstate support, but there is no promise of delivery of this functionality.
  • There is support for macOS 10.13 High Sierra on APFS on Splunk Enterprise version 7.1.

View solution in original post

Splunk Employee
Splunk Employee

I had the same issue on MacOS High Sierra after the upgrade from Sierra. You just need to had this line of code and it works. Tested with Splunk Enterprise 7.0

0 Karma

New Member

This worked for me..thank you.

0 Karma

Path Finder
0 Karma

New Member

Hey guys!!! SUPER new at splunk (tbh idk what I am doing)

I tried configuring $SPLUNK_HOME/etc/splunk-launch.conf: with OPTIMISTIC_ABOUT_FILE_LOCKING = 1. However, it keeps telling me $SPLUNK_HOME/etc/splunk-launch.conf: "No such file or directory". Super lost and frustrated with this, can someone break it down for a new comer?! I would really appreciated it.

0 Karma
State of Splunk Careers

Access the Splunk Careers Report to see real data that shows how Splunk mastery increases your value and job satisfaction.

Find out what your skills are worth!