Archive
Highlighted

Splunk unable to start and emits "Conf is currently being modified by process ####."

Explorer

This morning after rebooting my computer with splunk on it, Splunk refuses to start.

Trying to investigate the problem, I found a few odd things. The most likely error is identified by a message [Conf is currently being modified by process 4432] which occurs on a number of attempts at starting splunk, or trying to check licence via cli, for instance. The strange thing is that there does not seem to be a process 4432 running on my computer!

Has Splunk got corrupted somehow?

Here is an odd extract from the logs:
01-28-2013 02:30:55.412 +0800 INFO LicenseMgr - Initing LicenseMgr runContext_splunkd=false
01-28-2013 02:30:55.412 +0800 INFO LMStackMgr - closing stack mgr
01-28-2013 02:30:55.412 +0800 INFO LMSlaveInfo - all slaves cleared
01-28-2013 02:30:55.422 +0800 INFO LMStackMgr - created stack='download-trial'
01-28-2013 02:30:55.422 +0800 INFO LMStackMgr - have to auto-set active stack group='Trial' reason='invalid/missing group id' gidStr='' oldGid=Invalid

I start splunk via the CLI as:

"$ sudo /opt/splunk/bin/splunk start"

I then get the following:

"Splunk> Winning the War on Error

Checking prerequisites...
Checking http port [8000]: open
Checking mgmt port [8089]: open
Checking configuration... Done.
Checking indexes...
Validated databases: audit _blocksignature _internal _thefishbucket history main os sos sossummary_daily summary
Done
Checking filesystem compatibility... Done
Checking conf files for typos... Done
All preliminary checks passed.

Conf is currently being modified by process 4432.
Conf is currently being modified by process 4432.
Conf is currently being modified by process 4432.
Conf is currently being modified by process 4432.
Conf is currently being modified by process 4432.
Conf is currently being modified by process 4432.
Starting splunk server daemon (splunkd)...

Timed out waiting for splunkd to start.
Starting splunkweb... Done

If you get stuck, we're here to help.

Look for answers here: http://docs.splunk.com

The Splunk web interface is at http://wolfgang:8000"

and on attempting to reach splunk via the web interface, I get:

"The splunkd daemon cannot be reached by splunkweb. Check that there are no blocked network ports or that splunkd is still running."

With the following at the bottom of the screen:

"You are using wolfgang:8000, which is connected to splunkd @000 at https://127.0.0.1:8089 on Mon Jan 28 04:52:13 2013."

The @000 seems a bit odd, no?

Tags (3)
Highlighted

Re: Splunk unable to start and emits "Conf is currently being modified by process ####."

Champion

Sorry, just to be clear. What happens when you try to start Splunk and how are you starting Splunk?

0 Karma
Highlighted

Re: Splunk unable to start and emits "Conf is currently being modified by process ####."

Explorer

Thanks for your attention to this.

I edited my original post as comment on your question did not allow enough characters.

0 Karma
Highlighted

Re: Splunk unable to start and emits "Conf is currently being modified by process ####."

SplunkTrust
SplunkTrust

The @000 suggests that splunkweb isn't connected to splunkd, probably due to splunkd not having started.

0 Karma
Highlighted

Re: Splunk unable to start and emits "Conf is currently being modified by process ####."

Champion

when you say that there is no process 4432 running, how are you checking for this?

0 Karma
Highlighted

Re: Splunk unable to start and emits "Conf is currently being modified by process ####."

Explorer

I checked that there was no process 4432 doing a "ps aux".

Secondly, even after areboot it is still complaining about the same process 4432! It seems to me that Splunk has a variable set to 4432 somewhere that is persistent between reboots and restarts.

0 Karma
Highlighted

Re: Splunk unable to start and emits "Conf is currently being modified by process ####."

Splunk Employee
Splunk Employee

Very early in the life of conf-mutator.pid (5.x), the way the pid was tested would return true for any running THREAD. In modern linux, threads and processes IDs live in the same number space, so you can accidentally find a thread depending how you are testing for processes. You could have checked for a thread with ps auxH because.. H means .. tHread? or something?

0 Karma
Highlighted

Re: Splunk unable to start and emits "Conf is currently being modified by process ####."

Splunk Employee
Splunk Employee

I wonder if you do not have some file owned by a different user than the one running splunk

  • 1 stop splunk
  • 2 check the presence and owner/permissions on $SPLUNK_HOME/var/run/splunk/*.pid
  • 3 delete them if they still exists
  • 4 if needed do a chown -R for the splunk folders to change the owner
  • 5 start splunk under the correct user
  • 6 double check that the service starts with the correct user

View solution in original post

Highlighted

Re: Splunk unable to start and emits "Conf is currently being modified by process ####."

Splunk Employee
Splunk Employee

Hopefully, in 6.1.4+ / 6.2+ manually deleting pid files should not be necessary.
If it is, please do a little investigation of the system state, file contents, etc and file a bug.

0 Karma
Highlighted

Re: Splunk unable to start and emits "Conf is currently being modified by process ####."

Using 6.3.0, and manually deleting the conf-mutator.pid fixed the same problem for me.

0 Karma