Monitoring Splunk

Unable to start splunk in Indexer

BRG
Engager

I am having indexer clusters  & one of the indexer goes down due to some reason, I am unable to start splunk in that server. Its giving me below error.

[root@ bin]# ./splunk start
splunkd 21888 was not running.
Stopping splunk helpers...
                                                           [  OK  ]
Done.
Stopped helpers.
Removing stale pid file... Can't unlink pid file "/opt/splunk/var/run/splunk/splunkd.pid": Read-only file system
 
splunk version is 6.4.2 .
kindly help me to start splunk service.
Labels (3)
0 Karma

isoutamo
SplunkTrust
SplunkTrust

Hi

for reasons your file system has gone to read only mode. You must figure out why and then fix it by remounting it to rw-mode. After that you could star splunk as normal way. Then you must look are there any corrupted data or not. 
r.  Ismo

0 Karma

BRG
Engager

@isoutamoThanks for the info. I have checked file  permission  on other working indexers also, Its same only. Can you plz guide me how to find any corrupted data

0 Karma

isoutamo
SplunkTrust
SplunkTrust

Hi

I'm afraid that there is no any easy way to find it. Probably best options is to look from MC (monitoring console) that there is no buckets in unsync status (RF or SF is not fulfil).

Also try to look from internal logs that there is no ERROR level events related to indexing.

r. Ismo

0 Karma

shivanshu1593
Builder

You're restarting splunkd using root, instead of Splunk. This usually causes such problems. Are all of your Indexers running as root?

I'd try to check the permissions of the files, get rid of the PID and restart splunk using the user which was used to install the software, in most cases, it's Splunk.

Thank you,
Shiv
###If you found the answer helpful, kindly consider upvoting/accepting it as the answer as it helps other Splunkers find the solutions to similar issues###
0 Karma

isoutamo
SplunkTrust
SplunkTrust

If you are running splunk under systemd instead of traditional/old way, you actually must start it as root using the commend "systemctl start splunk.service" (or what ever your unit-file/service name is).  If you want still start it as splunk (or what ever your splunk service account is) you must add separately some additional tasks / rights to that user.

But if you are using it old way, then manage all starts alway as that service user like splunk, otherwise you has issues as @shivanshu1593 mentioned. But in your case there has been some OS level issues as filesystem has changed to readonly mode. And root cause for this is something which must figure first and then all other steps.

https://docs.splunk.com/Documentation/Splunk/8.0.5/Admin/ConfigureSplunktostartatboottime

r. Ismo

0 Karma

BRG
Engager

Sorry for the late reply,

Even though i tried to remove the file but its giving me error of Read-only file system.

rm: cannot remove `splunkd.pid': Read-only file system
 
Also moving file to another location is also not working.
Can we have any other option?
0 Karma

isoutamo
SplunkTrust
SplunkTrust
You must solve the reason why this file system has changed to read-only, fix it and there you must remount it to read write or even reboot the system and after that you can try to restart splunk, not earlier than fs is remounted to rw mode.
r. Ismo
0 Karma

impurush
Contributor

@BRG 

Looks like the splunk process is not able to read the file /opt/splunk/var/run/splunk/splunkd.pid.
Remove the splunkd.pid file under the location /opt/splunk/var/run/splunk and start again.

0 Karma

BRG
Engager
@impurush Will this change impact the other nodes in Indexer cluster ? As this is in production setup.

Also version of splunk is old.

0 Karma

impurush
Contributor

@BRG 

 By seeing ur question,It looks like the splunk is already stopped. So when you are starting you are getting this error right.

Also, removing this file will not affect your cluster.

To be safer side, check the Child process id in the file is already running or not. If not, u can kill the process id and  remove the file, then start the splunk.

0 Karma

BRG
Engager

@impurush Thanks for the info. file name conf-mutator.pid  have also same pid no. i.e 21888, do i have to remove this file also ?

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...