Installation

Unable to turn on splunk service on one of the indexer nodes

sdkp03
Path Finder

One of the indexer in production, is in shutdown state. While trying to start splunk service on this server, it fails with the following error message.

 

homePath='/dev/splunk/var/lib/splunk/audit/db' of index=_audit on unusable filesystem.

Validating databases (splunkd validatedb) failed with code '1'. If you cannot resolve the issue(s) above after consulting documentation, please file a case online at http://www.splunk.com/page/submit_issue

 

I did try to read through the troubleshooting article and even with “OPTIMISTIC_ABOUT_FILE_LOCKING=1” splunk service start up is still failing with same error.

Labels (1)
Tags (1)
0 Karma
1 Solution

sdkp03
Path Finder

Folder was accessible. Files were readable. However due to corrupt filesystem was not able to write anything in the said directory as splunk user which was the cause for the issue. Host was rebooted to fix the issue. 

View solution in original post

0 Karma

scelikok
SplunkTrust
SplunkTrust

Hi @sdkp03,

Did you check /dev/splunk/var/lib/splunk/audit/db folder accessible for the user splunkd process is using?

If this reply helps you an upvote is appreciated.

sdkp03
Path Finder

Folder was accessible. Files were readable. However due to corrupt filesystem was not able to write anything in the said directory as splunk user which was the cause for the issue. Host was rebooted to fix the issue. 

0 Karma
Get Updates on the Splunk Community!

Splunk Forwarders and Forced Time Based Load Balancing

Splunk customers use universal forwarders to collect and send data to Splunk. A universal forwarder can send ...

NEW! Log Views in Splunk Observability Dashboards Gives Context From a Single Page

Today, Splunk Observability releases log views, a new feature for users to add their logs data from Splunk Log ...

Last Chance to Submit Your Paper For BSides Splunk - Deadline is August 12th!

Hello everyone! Don't wait to submit - The deadline is August 12th! We have truly missed the community so ...