Monitoring Splunk

Splunk will not start and is waiting for config lock

mataharry
Communicator

My Splunk instance crashed and it won't restart with this error in splunkd.log:

01-30-2013 18:29:05.094 +0000 WARN loader - Sleep 10 sec, waiting for config lock. 
01-30-2013 18:29:15.094 +0000 WARN loader - Sleep 10 sec, waiting for config lock. 
01-30-2013 18:29:25.094 +0000 WARN loader - Sleep 10 sec, waiting for config lock. 
01-30-2013 18:29:35.094 +0000 WARN loader - Sleep 10 sec, waiting for config lock. 
01-30-2013 18:29:45.094 +0000 WARN loader - Sleep 10 sec, waiting for config lock. 
01-30-2013 18:29:55.094 +0000 WARN loader - Sleep 10 sec, waiting for config lock. 
01-30-2013 18:30:05.094 +0000 FATAL loader - Timed out waiting for config lock; see splunkd_stderr.log for details. Exiting.
Tags (2)
1 Solution

yannK
Splunk Employee
Splunk Employee

It's probably a locked file; to remove it:

  1. Stop splunk
  2. Remove any locked files
  3. Use this command ./splunk clean locks
  4. Or manually remove the file (*.pid or *.lock) in $SPLUNK_HOME\var\run\splunk\
  5. Start splunk and verify in the splunkd.log

View solution in original post

agrachevBLK
Explorer

In our case (6.2 Enterprise)

tail -f splunkd.log

suggested there was a lock on mongo.db

rm /opt/splunk/var/mongo/mongod.lock

did the job allowed the httpd process to start.

0 Karma

jrodman
Splunk Employee
Splunk Employee

For this "lock", it is actually a file called $SPLUNK_HOME/var/run/splunk/conf-mutator.pid, which really means something like "conf-modifying-program.pid".

Starting in 5.0, splunk.exe commands that modify the conf files determine whether splunkd is running or not, and ask running splunkd to modify the files for them if it is up. If it is not up, the splunk.exe command line program makes the changes itself. This file exists to synchronize the two possibilities, so we don't have files being edited by two programs at once.

The trouble comes in when something crashes, and leaves a stale conf-mutator.pid file lying around. If there is no program running anymore with that process number, it should cause no trouble (though in very early versions of 5.0.x we would detect threads with that number). Alternatively you can have a problem where that process number is running, but it's not splunk.exe or splunkd.exe. This latter problem is fixed in Splunk 6.1.3 and later.

marcoscala
Builder

I had a different error message, but the solution was the same:
Operation "ospath_fopen" failed in /opt/splunk/p4/splunk/branches/5.0.2/src/libzero/conf-mutator-locking.c:254, conf_mutator_lock(); Permission denied
Operation "ospath_fopen" failed in /opt/splunk/p4/splunk/branches/5.0.2/src/libzero/conf-mutator-locking.c:319, conf_mutator_unlock(); Permission denied

0 Karma

yannK
Splunk Employee
Splunk Employee

It's probably a locked file; to remove it:

  1. Stop splunk
  2. Remove any locked files
  3. Use this command ./splunk clean locks
  4. Or manually remove the file (*.pid or *.lock) in $SPLUNK_HOME\var\run\splunk\
  5. Start splunk and verify in the splunkd.log

jrodman
Splunk Employee
Splunk Employee

Very belatedly, the so-called "config lock" is $SPLUNK_HOME/var/run/splunk/conf-mutator.pid.

Simple version: 6.2.x+ should only show this behavior when it is correct for it to occur.

This message can occur correctly if you somehow try to run two splunkd programs at once, or if you have a splunk program running which is changing conf files while trying to start splunkd.

In versions before 6.1.3, the checking for running-pid was sloppy and would consider non-splunk programs to be valid owners of the pid.

If you are on such an old version, or if you see this on a new version and believe the behavior is working incorrectly:

  • Please cat the conf-mutator.pid file and check to see if any process is running with that process ID, to give us a hint about the problem to fix it (the behavior where early 6.1.x considers any program to be a match is the known bug).
  • Check to see whether you have any splunk or splunkd programs running. If there none around, and the splunk install is not on a shared filesystem, it should be safe to delete it. (6.2.x+ should do this automatically.)
  • On 6.2.x, for extra credit, please try: splunk cmd pid_check.sh conf-mutator pid_number_from_file . Then echo $? to see the exit code.

If not, it is safe to delete

0 Karma

w531t4
Path Finder

works for linux and windows

0 Karma

jdastmalchi_spl
Splunk Employee
Splunk Employee

you can use command ./splunk clean locks as an alternative to step2.

mataharry
Communicator

yep, I had a one of those in the folder, once removed it started.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...