Deployment Architecture

Not able to restart splunk instance

Contributor

Hi I am not able to restart splunk process

I am getting

Can't unlink pid file "/opt/splunk/var/run/splunk/splunkweb.pid": Read-only file system

error . can anyone help me?

0 Karma

Motivator

If you have a read-only file-system, and you have not set it that way, then you have bigger problems than not being able to start Splunk. It means the kernel has detected a filesystem inconsistency and to avoid compounding the corruption has switched the file-system to read-only mode. I am assuming that this is a Linux system (because that is a standard Linux error message to encounter and I don't believe - though nor do I know categorically - that Windows would issue that message).

With system administrative priveleges you should try interrogating the system error ring-buffer:

dmesg | less

Somewhere in that you will (should) have an error message detailing why the file-system was switched to read-only. The quickest, though disruptive, way for a novice to fix this is to reboot the system (ensuring you have the root password available) and allow the reboot process to attempt to correct the problem (with fsck). It will almost certainly encounter a problem which cannot be fixed automatically, which is where the root password comes in. You will be prompted to enter the system in maintenance mode, which requires that the root account is not disabled as a login account (which is possible, though uncommon) and that you have the password to authenticate with. If you reach this point the fix process will have given you a message explaining what you need to do. Unless you are an expert in Linux admin - and one assumes not, otherwise you would have got here by yourself - then the best course of action is to accept the default actions you are prompted with. This may cause data loss, but without a backup only some very intricate deep-level repair work can hope to stop that, and that requires a lot of experience.

Motivator

Please signify acceptance of the answer if you found it helpful.

0 Karma

Contributor

Talked to admin team and seems like filesystem issue.Thanks for help.

0 Karma

Motivator

It is possible that the file-system corruption is the result of a failing hardware component (disk or memory, most likely), and you may well find yourself facing the same problem again.

SplunkTrust
SplunkTrust

Hi adityapavan18

well according to the message the user trying to restart splunk does not have permission on the file /opt/splunk/var/run/splunk/splunkweb.pid. If your file system would be read only, your splunk would have thrown many, many other messages nor would be able to index any kind of data.

cheers, MuS

SplunkTrust
SplunkTrust

okay like always got curious and did some testing:
permission errors will be reported like this

Pid file "/opt/splunk/var/run/splunk/splunkweb.pid" unreadable.: Permission denied

so for this part my answer is wrong. But like grijhwani and I did say correctly, having the file system in read-only mounted will show more errors and possible end in data loss.

Ultra Champion

We are facing the exact same problem -

when starting it - 

Done.
Stopped helpers.
Removing stale pid file... Can't unlink pid file "/opt/splunk/splunkforwarder/var/run/splunk/splunkd.pid": Read-only file system



$ ll
total 36
drwx--x--x. 4 splnkfwd splnkfwd 4096 Jun 24  2015 appserver
-rw-------. 1 splnkfwd splnkfwd 9388 Aug  5 21:08 composite.xml
-rw-------. 1 splnkfwd splnkfwd    6 Aug  5 21:08 conf-mutator.pid
drwx--x--x. 2 splnkfwd splnkfwd 4096 Oct 14  2015 dispatch
drwx------. 2 splnkfwd splnkfwd 4096 Aug  5 21:08 merged
-rw-r-----. 1 splnkfwd splnkfwd   12 Aug  5 21:08 splunkd.pid
drwx------. 2 splnkfwd splnkfwd 4096 Jun 24  2015 upload
splnkfwd@xxxxxxx:/opt/splunk/splunkforwarder/var/run/splunk
$ rm splunkd.pid
rm: cannot remove `splunkd.pid': Read-only file system
splnkfwd@xxxxxx:/opt/splunk/splunkforwarder/var/run/splunk
0 Karma

Champion
0 Karma

Motivator

I downvoted this post because that is not a solution. It requires that the pid file be removable. The process you link to will fail because the file-system has been set to read-only by the kernel, a situation which implies a significant system fault which needs to be addressed. The very problem being explicitly reported by the start-up is that the pid file cannot be removed.

0 Karma

Motivator

This is a 3 year-old question. You should, really, if you want to garner a new answer, open a new question.

That is largely irrelevant, though. As it is all of my previous responses stand. There's nothing new or different about your problem that isn't already addressed above. The file-system in read-only mode, when to have written the PID file it must have previously have been read-write suggests a file-system error was detected by the kernel. You need to check your logs and dmesg buffer for disk and file-system errors and remediate the problem. Until you do, the system is highly suspect and at risk. You may want to consider backing it up before you do, but you're going to need to reboot (with the understanding that if this is the first manifestation of a bigger fault, it may not come back). Read my answers above. This isn't a Splunk problem, it's a fundamental sysadmin problem.

0 Karma

Champion

I believe the error doesn't have anything to do with the splunk restart process and MUS is correct with his remark.Seems like permission issue. Let's wait for the person who raised this question.

0 Karma

Motivator

No - according to the error message the file-system as a whole is read only. It is quite explicit. If it were a user permission issue, the message would say so.

0 Karma