Getting Data In

Why am I getting "homePath='/opt/splunk/var/lib/splunk/audit/db' of index=_audit on unusable filesystem." while starting Splunk on an indexer?

ayushchoudhary
Path Finder

I got this error while starting Splunk on the indexer.

homePath='/opt/splunk/var/lib/splunk/audit/db' of index=_audit on unusable filesystem. Validating databases (splunkd validatedb) failed with code '1'. 

Please help urgently.

1 Solution

naisanza
Path Finder

You'll want to append the following configuration option to $SPLUNK_HOME/etc/splunk-launch.conf:

OPTIMISTIC_ABOUT_FILE_LOCKING = 1

Note from malmoore (Splunk): As of 28 March 2018, this workaround has been officially documented in the Troubleshooting Manual. See Splunk Enterprise does not start due to unusable filesystem in the manual for the procedure.

The caveats for using this workaround still apply. Proceed with caution, and at your own risk. Irrevocable data loss can still occur. We have already had one report in this thread of problems that have occurred after enabling this setting.

Key points (as of 24 April 2018)

  • There is still no support for macOS 10.13 High Sierra on Splunk Enterprise version 7.0.
  • There is work scheduled to fix the problem for macOS 10.13 on Splunk Enterprise 7.0 and reinstate support, but there is no promise of delivery of this functionality.
  • There is support for macOS 10.13 High Sierra on APFS on Splunk Enterprise version 7.1.

View solution in original post

jmantor
Path Finder

I just encountered this same error running Splunk 6.5.6 on RHEL with an EXT4 file system.

0 Karma

malmoore
Splunk Employee
Splunk Employee

Is this a fresh install?
What version of RHEL?
Did you upgrade and switch from another file system to ext4?

0 Karma

jmantor
Path Finder

It was an existing install. RHEL 6.x. It turns out the lun that the disk was on was accidentally filled up via a VMware snapshot.

0 Karma

malmoore
Splunk Employee
Splunk Employee

Ah, thanks or the info. Another reason why setting this variable should be done only as a last resort.

0 Karma

bnariyani_splun
Splunk Employee
Splunk Employee

Worked Well...

0 Karma

rcrohns_splunk
Splunk Employee
Splunk Employee

This worked for me on macOS High Sierra 10.13.3 with Splunk version 7.0.2.,

uthornander_spl
Splunk Employee
Splunk Employee

Is this still applicable to 7.1?

UT
0 Karma

sloshburch
Splunk Employee
Splunk Employee

Did you mean Splunk Enterprise 7.0.1 or an OS version?

0 Karma

uthornander_spl
Splunk Employee
Splunk Employee

Is this still applicable to 7.1?

UT
0 Karma

tjr1775
Path Finder

So I had this problem as well, and the optimistic file thingee =1 did not work. I have MacOS 10.13.1 (High Sierra) and Splunk 7.0. It was thought this wasn't a problem with 7.0, but it is. However, here is the fix if the optimistic thing is well, not so optimistic:

rm /opt/splunk/lib/libz.1.dylib
cp /usr/lib/libz.1.dylib /opt/splunk/lib/libz.1.dylib

Found at a similar thread: https://answers.splunk.com/answers/585512/importerror-symbol-not-found-inflatevalidate-when.html

pranavna
Explorer

that fixed my issue. thanks...

0 Karma

s2_splunk
Splunk Employee
Splunk Employee

Careful. This is an indication that you may have Splunk deployed on top of an unsupported filesystem that does not implement required file locking mechanism. Setting that attribute in splunk-launch.conf is overriding our internal file locking test during startup. YMMV...

vbumgarner
Contributor

After using this flag for awhile, I'm now getting:

WARN JournalSlice - Error reading from fresh journal slice file ".../db/hot_v1_4937/rawdata/1971039751": Input/output error

Is this related, or do I just have a bad disk?

0 Karma

Dimitri_McKay
Splunk Employee
Splunk Employee

Actually, this worked perfect for me too. I'm running Sierra beta and two of my instances complained about this. So it may be OS X Beta related.

0 Karma

vbumgarner
Contributor

This is still required on the public release of Sierra.

0 Karma

naisanza
Path Finder

You'll want to append the following configuration option to $SPLUNK_HOME/etc/splunk-launch.conf:

OPTIMISTIC_ABOUT_FILE_LOCKING = 1

Note from malmoore (Splunk): As of 28 March 2018, this workaround has been officially documented in the Troubleshooting Manual. See Splunk Enterprise does not start due to unusable filesystem in the manual for the procedure.

The caveats for using this workaround still apply. Proceed with caution, and at your own risk. Irrevocable data loss can still occur. We have already had one report in this thread of problems that have occurred after enabling this setting.

Key points (as of 24 April 2018)

  • There is still no support for macOS 10.13 High Sierra on Splunk Enterprise version 7.0.
  • There is work scheduled to fix the problem for macOS 10.13 on Splunk Enterprise 7.0 and reinstate support, but there is no promise of delivery of this functionality.
  • There is support for macOS 10.13 High Sierra on APFS on Splunk Enterprise version 7.1.

im_bharath
Path Finder

Hey @naisanza

I installed the splunk on the Ubuntu (WSL) and encountered the same issue and i have tried the option you have provided and it worked.. Thank you very much. 

0 Karma

ipapa_splunk
Splunk Employee
Splunk Employee

I had the same issue on MacOS High Sierra after the upgrade from Sierra. You just need to had this line of code and it works. Tested with Splunk Enterprise 7.0

0 Karma

nitinkul
New Member

This worked for me..thank you.

0 Karma

Jason_S
Path Finder
0 Karma
Get Updates on the Splunk Community!

Exporting Splunk Apps

Join us on Monday, October 21 at 11 am PT | 2 pm ET!With the app export functionality, app developers and ...

Cisco Use Cases, ITSI Best Practices, and More New Articles from Splunk Lantern

Splunk Lantern is a Splunk customer success center that provides advice from Splunk experts on valuable data ...

Build Your First SPL2 App!

Watch the recording now!.Do you want to SPL™, too? SPL2, Splunk's next-generation data search and preparation ...