Not able to restart splunk instance

[adityapavan18]

Hi I am not able to restart splunk process

I am getting

Can't unlink pid file "/opt/splunk/var/run/splunk/splunkweb.pid": Read-only file system

error . can anyone help me?

[grijhwani]

If you have a read-only file-system, and you have not set it that way, then you have bigger problems than not being able to start Splunk. It means the kernel has detected a filesystem inconsistency and to avoid compounding the corruption has switched the file-system to read-only mode. I am assuming that this is a Linux system (because that is a standard Linux error message to encounter and I don't believe - though nor do I know categorically - that Windows would issue that message).

With system administrative priveleges you should try interrogating the system error ring-buffer:

dmesg | less

Somewhere in that you will (should) have an error message detailing why the file-system was switched to read-only. The quickest, though disruptive, way for a novice to fix this is to reboot the system (ensuring you have the root password available) and allow the reboot process to attempt to correct the problem (with fsck). It will almost certainly encounter a problem which cannot be fixed automatically, which is where the root password comes in. You will be prompted to enter the system in maintenance mode, which requires that the root account is not disabled as a login account (which is possible, though uncommon) and that you have the password to authenticate with. If you reach this point the fix process will have given you a message explaining what you need to do. Unless you are an expert in Linux admin - and one assumes not, otherwise you would have got here by yourself - then the best course of action is to accept the default actions you are prompted with. This may cause data loss, but without a backup only some very intricate deep-level repair work can hope to stop that, and that requires a lot of experience.

[grijhwani]

Please signify acceptance of the answer if you found it helpful.

[adityapavan18]

Talked to admin team and seems like filesystem issue.Thanks for help.

[grijhwani]

It is possible that the file-system corruption is the result of a failing hardware component (disk or memory, most likely), and you may well find yourself facing the same problem again.

[MuS]

Hi adityapavan18

well according to the message the user trying to restart splunk does not have permission on the file /opt/splunk/var/run/splunk/splunkweb.pid. If your file system would be read only, your splunk would have thrown many, many other messages nor would be able to index any kind of data.

cheers, MuS

[MuS]

okay like always got curious and did some testing:
permission errors will be reported like this

Pid file "/opt/splunk/var/run/splunk/splunkweb.pid" unreadable.: Permission denied

so for this part my answer is wrong. But like grijhwani and I did say correctly, having the file system in read-only mounted will show more errors and possible end in data loss.

[ddrillic]

We are facing the exact same problem -

when starting it - 

Done.
Stopped helpers.
Removing stale pid file... Can't unlink pid file "/opt/splunk/splunkforwarder/var/run/splunk/splunkd.pid": Read-only file system



$ ll
total 36
drwx--x--x. 4 splnkfwd splnkfwd 4096 Jun 24  2015 appserver
-rw-------. 1 splnkfwd splnkfwd 9388 Aug  5 21:08 composite.xml
-rw-------. 1 splnkfwd splnkfwd    6 Aug  5 21:08 conf-mutator.pid
drwx--x--x. 2 splnkfwd splnkfwd 4096 Oct 14  2015 dispatch
drwx------. 2 splnkfwd splnkfwd 4096 Aug  5 21:08 merged
-rw-r-----. 1 splnkfwd splnkfwd   12 Aug  5 21:08 splunkd.pid
drwx------. 2 splnkfwd splnkfwd 4096 Jun 24  2015 upload
splnkfwd@xxxxxxx:/opt/splunk/splunkforwarder/var/run/splunk
$ rm splunkd.pid
rm: cannot remove `splunkd.pid': Read-only file system
splnkfwd@xxxxxx:/opt/splunk/splunkforwarder/var/run/splunk

[linu1988]

There is a solution to it

https://answers.splunk.com/answers/73843/splunk-will-not-start-and-is-waiting-for-config-lock.html

[grijhwani]

I downvoted this post because that is not a solution. It requires that the pid file be removable. The process you link to will fail because the file-system has been set to read-only by the kernel, a situation which implies a significant system fault which needs to be addressed. The very problem being explicitly reported by the start-up is that the pid file cannot be removed.

[grijhwani]

This is a 3 year-old question. You should, really, if you want to garner a new answer, open a new question.

That is largely irrelevant, though. As it is all of my previous responses stand. There's nothing new or different about your problem that isn't already addressed above. The file-system in read-only mode, when to have written the PID file it must have previously have been read-write suggests a file-system error was detected by the kernel. You need to check your logs and dmesg buffer for disk and file-system errors and remediate the problem. Until you do, the system is highly suspect and at risk. You may want to consider backing it up before you do, but you're going to need to reboot (with the understanding that if this is the first manifestation of a bigger fault, it may not come back). Read my answers above. This isn't a Splunk problem, it's a fundamental sysadmin problem.

[linu1988]

I believe the error doesn't have anything to do with the splunk restart process and MUS is correct with his remark.Seems like permission issue. Let's wait for the person who raised this question.

[grijhwani]

No - according to the error message the file-system as a whole is read only. It is quite explicit. If it were a user permission issue, the message would say so.