I am having indexer clusters & one of the indexer goes down due to some reason, I am unable to start splunk in that server. Its giving me below error.
Hi
for reasons your file system has gone to read only mode. You must figure out why and then fix it by remounting it to rw-mode. After that you could star splunk as normal way. Then you must look are there any corrupted data or not.
r. Ismo
@isoutamoThanks for the info. I have checked file permission on other working indexers also, Its same only. Can you plz guide me how to find any corrupted data
Hi
I'm afraid that there is no any easy way to find it. Probably best options is to look from MC (monitoring console) that there is no buckets in unsync status (RF or SF is not fulfil).
Also try to look from internal logs that there is no ERROR level events related to indexing.
r. Ismo
You're restarting splunkd using root, instead of Splunk. This usually causes such problems. Are all of your Indexers running as root?
I'd try to check the permissions of the files, get rid of the PID and restart splunk using the user which was used to install the software, in most cases, it's Splunk.
If you are running splunk under systemd instead of traditional/old way, you actually must start it as root using the commend "systemctl start splunk.service" (or what ever your unit-file/service name is). If you want still start it as splunk (or what ever your splunk service account is) you must add separately some additional tasks / rights to that user.
But if you are using it old way, then manage all starts alway as that service user like splunk, otherwise you has issues as @shivanshu1593 mentioned. But in your case there has been some OS level issues as filesystem has changed to readonly mode. And root cause for this is something which must figure first and then all other steps.
https://docs.splunk.com/Documentation/Splunk/8.0.5/Admin/ConfigureSplunktostartatboottime
r. Ismo
Sorry for the late reply,
Even though i tried to remove the file but its giving me error of Read-only file system.
@BRG
Looks like the splunk process is not able to read the file /opt/splunk/var/run/splunk/splunkd.pid.
Remove the splunkd.pid file under the location /opt/splunk/var/run/splunk and start again.
Also version of splunk is old.
By seeing ur question,It looks like the splunk is already stopped. So when you are starting you are getting this error right.
Also, removing this file will not affect your cluster.
To be safer side, check the Child process id in the file is already running or not. If not, u can kill the process id and remove the file, then start the splunk.
@impurush Thanks for the info. file name conf-mutator.pid have also same pid no. i.e 21888, do i have to remove this file also ?