topic Re: Splunk will not start and is waiting for config lock in Monitoring Splunk

Splunk will not start and is waiting for config lock

mataharry — Wed, 30 Jan 2013 19:14:24 GMT

My Splunk instance crashed and it won't restart with this error in splunkd.log:

01-30-2013 18:29:05.094 +0000 WARN loader - Sleep 10 sec, waiting for config lock. 
01-30-2013 18:29:15.094 +0000 WARN loader - Sleep 10 sec, waiting for config lock. 
01-30-2013 18:29:25.094 +0000 WARN loader - Sleep 10 sec, waiting for config lock. 
01-30-2013 18:29:35.094 +0000 WARN loader - Sleep 10 sec, waiting for config lock. 
01-30-2013 18:29:45.094 +0000 WARN loader - Sleep 10 sec, waiting for config lock. 
01-30-2013 18:29:55.094 +0000 WARN loader - Sleep 10 sec, waiting for config lock. 
01-30-2013 18:30:05.094 +0000 FATAL loader - Timed out waiting for config lock; see splunkd_stderr.log for details. Exiting.

Re: Splunk will not start and is waiting for config lock

yannK — Wed, 30 Jan 2013 19:16:20 GMT

It's probably a locked file; to remove it:

Stop splunk
Remove any locked files
Use this command ./splunk clean locks
Or manually remove the file (*.pid or *.lock) in $SPLUNK_HOME\var\run\splunk\
Start splunk and verify in the splunkd.log

Re: Splunk will not start and is waiting for config lock

mataharry — Wed, 30 Jan 2013 19:17:48 GMT

yep, I had a one of those in the folder, once removed it started.

Re: Splunk will not start and is waiting for config lock

marcoscala — Mon, 28 Sep 2020 13:51:56 GMT

I had a different error message, but the solution was the same:
Operation "ospath_fopen" failed in /opt/splunk/p4/splunk/branches/5.0.2/src/libzero/conf-mutator-locking.c:254, conf_mutator_lock(); Permission denied
Operation "ospath_fopen" failed in /opt/splunk/p4/splunk/branches/5.0.2/src/libzero/conf-mutator-locking.c:319, conf_mutator_unlock(); Permission denied

Re: Splunk will not start and is waiting for config lock

jdastmalchi_spl — Fri, 15 Nov 2013 18:21:26 GMT

you can use command ./splunk clean locks as an alternative to step2.

Re: Splunk will not start and is waiting for config lock

w531t4 — Mon, 18 Nov 2013 18:15:08 GMT

works for linux and windows

Re: Splunk will not start and is waiting for config lock

jrodman — Fri, 14 Nov 2014 15:31:51 GMT

For this "lock", it is actually a file called $SPLUNK_HOME/var/run/splunk/conf-mutator.pid, which really means something like "conf-modifying-program.pid".

Starting in 5.0, splunk.exe commands that modify the conf files determine whether splunkd is running or not, and ask running splunkd to modify the files for them if it is up. If it is not up, the splunk.exe command line program makes the changes itself. This file exists to synchronize the two possibilities, so we don't have files being edited by two programs at once.

The trouble comes in when something crashes, and leaves a stale conf-mutator.pid file lying around. If there is no program running anymore with that process number, it should cause no trouble (though in very early versions of 5.0.x we would detect threads with that number). Alternatively you can have a problem where that process number is running, but it's not splunk.exe or splunkd.exe. This latter problem is fixed in Splunk 6.1.3 and later.

Re: Splunk will not start and is waiting for config lock

agrachevBLK — Thu, 13 Aug 2015 02:28:50 GMT

In our case (6.2 Enterprise)

tail -f splunkd.log

suggested there was a lock on mongo.db

rm /opt/splunk/var/mongo/mongod.lock

did the job allowed the httpd process to start.

Re: Splunk will not start and is waiting for config lock

jrodman — Tue, 29 Sep 2020 08:29:33 GMT

Very belatedly, the so-called "config lock" is $SPLUNK_HOME/var/run/splunk/conf-mutator.pid.

Simple version: 6.2.x+ should only show this behavior when it is correct for it to occur.

This message can occur correctly if you somehow try to run two splunkd programs at once, or if you have a splunk program running which is changing conf files while trying to start splunkd.

In versions before 6.1.3, the checking for running-pid was sloppy and would consider non-splunk programs to be valid owners of the pid.

If you are on such an old version, or if you see this on a new version and believe the behavior is working incorrectly:

Please cat the conf-mutator.pid file and check to see if any process is running with that process ID, to give us a hint about the problem to fix it (the behavior where early 6.1.x considers any program to be a match is the known bug).
Check to see whether you have any splunk or splunkd programs running. If there none around, and the splunk install is not on a shared filesystem, it should be safe to delete it. (6.2.x+ should do this automatically.)
On 6.2.x, for extra credit, please try: splunk cmd pid_check.sh conf-mutator pid_number_from_file . Then echo $? to see the exit code.

If not, it is safe to delete