Why am I getting "homePath='/opt/splunk/var/lib/splunk/audit/db' of index=_audit on unusable filesystem." while starting Splunk on an indexer?

[ayushchoudhary]

I got this error while starting Splunk on the indexer.

homePath='/opt/splunk/var/lib/splunk/audit/db' of index=_audit on unusable filesystem. Validating databases (splunkd validatedb) failed with code '1'. 

Please help urgently.

[DanKneeVee]

Hey guys!!! SUPER new at splunk (tbh idk what I am doing)

I tried configuring $SPLUNK_HOME/etc/splunk-launch.conf: with OPTIMISTIC_ABOUT_FILE_LOCKING = 1. However, it keeps telling me $SPLUNK_HOME/etc/splunk-launch.conf: "No such file or directory". Super lost and frustrated with this, can someone break it down for a new comer?! I would really appreciated it.

[kmugglet]

Hi Dan.
Not sure if you got this sorted.
Just check that your $SPLUNK_HOME is set 

otherwise on linux try editing /opt/splunk/etc/splunk-launch.conf

[DanKneeVee]

Hey guys I was so overwhelmed while trying to get answers I skimmed through @naisanza 's link (Splunk Enterprise does not start due to unusable filesystem). And fixing the problem looks fairly simple. HOWEVER, the manual does warn me that if I "bypass filesystem checks" then "irrevocable data loss can occur".
This leads me into a three part question;
1) What does this mean? Could someone explain what "bypass filesystem checks" and what "irrevocable data loss" is?
2) Should I be concerned about doing this? I am in no way a professional in the field, I just heard this program (Splunk) was used for cyber security. So losing some data would be okay/not okay in my case?
3) As a novice in the cyber security field, should I even be bothering with Splunk? I have plans on doing the Splunk Bootcamps btw. I am really passionate on learning about programs like Splunk and others.

Thanks, I will be in debt for your help and guidance.

[malmoore]

Hi,

To answer your question about the setting: Turning it on means that Splunk no longer attempts to check and see if it can run on your filesystem before starting and indexing data. That means, if your filesystem doesn't do the right things when Splunk attempts to write index buckets, you could potentially lose any data that is contained within those buckets if Splunk can't retrieve them later. This is why it is so important to let Splunk check the filesystem before it starts, because if it doesn't, that means it doesn't understand the layout of the filesystem and is protecting you from that potential data loss.

Like the article says, if you understand these risks and want to index your data anyway, go right ahead. If you have additional copies of the data, or can reindex the data that you have, then there is no real danger. But you should do what you can to make sure that Splunk runs on the filesystem that is on your machine, that means using a supported filesystem.

Of course I'm going to say, use Splunk all day every day for everything 😄 But the answer to that question is dependent on your specific needs. This is a temporary solution to a fairly uncommon problem which can easily be fixed by using Splunk on a supported filesystem on a supported operating system. You should never encounter this problem during normal Splunk usage.

[sloshburch]

What keeps telling you "No such file or directory"? Show us the command you run that produces that output? @DanKneeVee

[malmoore]

Sounds like he's using the literal $SPLUNK_HOME/etc/splunk-launch.conf rather than the actual location of the file.

@DanKneeVee, where is your Splunk installation at? If it is in, for example, /Applications/splunk, then you would edit /Applications/splunk/etc/splunk-launch.conf. If it is in /opt/splunk, then you would edit /opt/splunk/etc/splunk-launch.conf.

[koshyk]

Any idea how to add in docker splunk?

I tried putting it in SPLUNK_BEFORE_START_CMD_xx but Seems the splunk-lauch.conf is created at time of start. So getting wiped off.
Adding as SPLUNK_CMD not worthy as the failure comes just after start.

[sloshburch]

@koshyk - are you aware of https://hub.docker.com/r/splunk/splunk/

[rithvikmundra]

Works perfectly fine for me. Thank you!

[antoniocheltenh]

Worked for my High Serria Splunk 7.x

[mohdeee]

Thanks, it helped 🙂

[malmoore]

I would like to remind everyone here that this is NOT a long term solution for this problem. By setting this variable you are defeating filesystem locking checks and thus potentially rendering any data that you index inaccessible later. Just because the error message goes away doesn't mean it's working! We have already had one report in this thread of problems that have occurred after enabling this setting. You're free to do this but please understand that you do it solely at your own risk. There is work scheduled to fix the error for macOS 10.13, but I can't promise a timeframe or offer further details on when that work will be completed. Proceed with caution.

[Esky73]

FYI - You also get this error if you run splunk in the Ubuntu Windows Subsystem for Linux

and this workaround works there also.

Splunk 7.0.1

[Aguilar7]

where is this line being added?

[yannK]

it should be added to $SPLUNK_HOME/etc/splunk-launch.conf

[tjr1775]

Adding the line to splunk-launch.conf did not work for me running High Sierra 10.13.1 and Splunk 7.0. Still getting this error:

Checking prerequisites...
    Checking http port [8000]: open
    Checking mgmt port [8089]: open
    Checking appserver port [127.0.0.1:8065]: open
    Checking kvstore port [8191]: open
Traceback (most recent call last):
  File "/Applications/Splunk/lib/python2.7/site-packages/splunk/clilib/cli.py", line 17, in 
    import splunk.clilib.cli_common as comm
  File "/Applications/Splunk/lib/python2.7/site-packages/splunk/clilib/cli_common.py", line 10, in 
    from xml.sax import saxutils
  File "/Applications/Splunk/lib/python2.7/xml/sax/saxutils.py", line 6, in 
    import os, urlparse, urllib, types
  File "/Applications/Splunk/lib/python2.7/urllib.py", line 1440, in 
    from _scproxy import _get_proxy_settings, _get_proxies
ImportError: dlopen(/Applications/Splunk/lib/python2.7/lib-dynload/_scproxy.so, 2): Symbol not found: _inflateValidate
  Referenced from: /System/Library/Frameworks/ImageIO.framework/Versions/A/Resources/libPng.dylib
  Expected in: /Applications/Splunk/lib/libz.1.dylib
 in /System/Library/Frameworks/ImageIO.framework/Versions/A/Resources/libPng.dylib

[tjr1775]

disregard- I answered the original question with the fix

[sloshburch]

@tjr1775 - You may be able to delete your prior post if you feel it doesn't apply. You might have options that appear when you hover by your name in the comment, click the gear, and select to delete.

[tcccorp]

Hello,

I had the same issue with the latest version under ubuntu 16.04 ( I used the tgz version) . I added the line on the conf file and now it works !

Thanks fir this help

[aharkare]

After adding 'OPTIMISTIC_ABOUT_FILE_LOCKING = 1' in $SPLUNK_HOME/etc/splunk-launch.conf, it worked for Mac OS High Sierra 10.13.1 (17B48).

Thank you so much for the suggestion!

[potap75]

Same here! Thank you!