topic Re: Unable to start splunk in Indexer in Monitoring Splunk

Unable to start splunk in Indexer

BRG — Tue, 11 Aug 2020 03:25:14 GMT

I am having indexer clusters & one of the indexer goes down due to some reason, I am unable to start splunk in that server. Its giving me below error.

[root@ bin]# ./splunk start
splunkd 21888 was not running.
Stopping splunk helpers...
[ OK ]
Done.
Stopped helpers.
Removing stale pid file... Can't unlink pid file "/opt/splunk/var/run/splunk/splunkd.pid": Read-only file system

splunk version is 6.4.2 .

kindly help me to start splunk service.

Re: Unable to start splunk in Indexer

impurush — Tue, 11 Aug 2020 03:53:48 GMT

@BRG

Looks like the splunk process is not able to read the file /opt/splunk/var/run/splunk/splunkd.pid.
Remove the splunkd.pid file under the location /opt/splunk/var/run/splunk and start again.

Re: Unable to start splunk in Indexer

BRG — Tue, 11 Aug 2020 04:46:29 GMT

@impurush Will this change impact the other nodes in Indexer cluster ? As this is in production setup.

Also version of splunk is old.

Re: Unable to start splunk in Indexer

impurush — Tue, 11 Aug 2020 04:56:04 GMT

@BRG

By seeing ur question,It looks like the splunk is already stopped. So when you are starting you are getting this error right.

Also, removing this file will not affect your cluster.

To be safer side, check the Child process id in the file is already running or not. If not, u can kill the process id and remove the file, then start the splunk.

Re: Unable to start splunk in Indexer

isoutamo — Tue, 11 Aug 2020 05:03:07 GMT

for reasons your file system has gone to read only mode. You must figure out why and then fix it by remounting it to rw-mode. After that you could star splunk as normal way. Then you must look are there any corrupted data or not.
r. Ismo

Re: Unable to start splunk in Indexer

BRG — Tue, 11 Aug 2020 06:39:20 GMT

@isoutamoThanks for the info. I have checked file permission on other working indexers also, Its same only. Can you plz guide me how to find any corrupted data

Re: Unable to start splunk in Indexer

BRG — Tue, 11 Aug 2020 06:42:27 GMT

@impurush Thanks for the info. file name conf-mutator.pid have also same pid no. i.e 21888, do i have to remove this file also ?

Re: Unable to start splunk in Indexer

isoutamo — Tue, 11 Aug 2020 06:42:42 GMT

I'm afraid that there is no any easy way to find it. Probably best options is to look from MC (monitoring console) that there is no buckets in unsync status (RF or SF is not fulfil).

Also try to look from internal logs that there is no ERROR level events related to indexing.

r. Ismo

Re: Unable to start splunk in Indexer

shivanshu1593 — Wed, 12 Aug 2020 22:08:05 GMT

You're restarting splunkd using root, instead of Splunk. This usually causes such problems. Are all of your Indexers running as root?

I'd try to check the permissions of the files, get rid of the PID and restart splunk using the user which was used to install the software, in most cases, it's Splunk.

Re: Unable to start splunk in Indexer

isoutamo — Thu, 13 Aug 2020 08:14:33 GMT

If you are running splunk under systemd instead of traditional/old way, you actually must start it as root using the commend "systemctl start splunk.service" (or what ever your unit-file/service name is). If you want still start it as splunk (or what ever your splunk service account is) you must add separately some additional tasks / rights to that user.

But if you are using it old way, then manage all starts alway as that service user like splunk, otherwise you has issues as @shivanshu1593 mentioned. But in your case there has been some OS level issues as filesystem has changed to readonly mode. And root cause for this is something which must figure first and then all other steps.

https://docs.splunk.com/Documentation/Splunk/8.0.5/Admin/ConfigureSplunktostartatboottime

r. Ismo

Re: Unable to start splunk in Indexer

BRG — Thu, 17 Sep 2020 07:00:18 GMT

Sorry for the late reply,

Even though i tried to remove the file but its giving me error of Read-only file system.

rm: cannot remove `splunkd.pid': Read-only file system

Also moving file to another location is also not working.

Can we have any other option?

Re: Unable to start splunk in Indexer

isoutamo — Thu, 17 Sep 2020 08:41:46 GMT

You must solve the reason why this file system has changed to read-only, fix it and there you must remount it to read write or even reboot the system and after that you can try to restart splunk, not earlier than fs is remounted to rw mode.
r. Ismo