RF and SF are not meeting up

[sandeep_A1997]

Suddenly we observed /opt/data was unmounted, and ownership has changed from splunk to root. Mounted back and restarted the service. still SF and RF are not meeting up. Restarted the service from AWS, still no response, we have 3 indexers placed in this cluster.

tried rollingg restart for remaining indexers, when i restarted the second indexer, the splunk stopped and /opt/data  ownership changed and unmounted, mounted them again same happend with 1st indexer too, didnot touched 3rd indexer.

Now amoung 3 indexer 2 were down restarted then and started splunk in them and mounted /opt/data too, still we are not able to see SF and RF are meeting.

[livehybrid]

Hi @sandeep_A1997 

Can you confirm that the Indexer itself hasnt restarted at an OS level? This might explain why the data volume unmounted. What is the uptime on the indexer?

The unmounting and permission changing is something which must be happening outside of Splunk, so its important to get to the bottom of what is causing this, it could be that the host crashed and rebooted or something on the AWS side (e.g. automations).

I would recommend ensuring Splunk is stopped on an indexer, then run

sudo chown -R splunk:splunk /opt/data

This will recurisvely change the ownership to splunk. Once this is done start up the Splunk service, repeat this on the other faulty indexer(s).

After some time the indexers should all be back up and the cluster manager should do its job to repair the cluster.

🌟 Did this answer help you? If so, please consider:

• Adding karma to show it was useful
• Marking it as the solution if it resolved your issue
• Commenting if you need any clarification

Your feedback encourages the volunteers in this community to continue contributing

[isoutamo]

You should also check if CM see those peers as member of indexer cluster. Then also check what errors and maybe warnings which told what has happened. 

[inventsekar]

Hi @sandeep_A1997 

Pls check the bucket status - indexer clustering > Indexes > Bucket Status

Pls update us if you have any bucket issues... 

Some docs links:

https://help.splunk.com/en/splunk-enterprise/administer/manage-indexers-and-indexer-clusters/9.4/tro...

https://splunk.my.site.com/customer/s/article/SF-and-RF-is-not-met-on-Cluster-Manager