Distsearch.conf and other config files overridden after rolling retsart

[neeravmathur]

Hi, 

We have 3 search head in a cluster and 3 indexers in non clustered environment. Whenever we do a rolling restart of the SH, the distsearch.conf in etc/system/local and some lookup csv in some of apps change. It does not happen always but very often. Can anyone help in figuring why this happens and what needs to be corrected. There is no other distsearch.conf anywhere on the SH.

Thanks for your help....

[scelikok]

Hi @neeravmathur,

So, I think one of your search heads in the cluster cannot sync with the captain. You can see below document for sync problems;

https://docs.splunk.com/Documentation/Splunk/8.1.2/DistSearch/HowconfrepoworksinSHC#Replication_sync... 

Distributing app via deployer is best practice but I don't think it will help since it will update the files only after apply shcluster-bundle command. The reason I asked this is, a member would have been rejoins to cluster and sync. 

[scelikok]

Hi @neeravmathur,

You can check the distsearch.conf file and lookups in your deployer. 

[neeravmathur]

Hi @scelikok, 

Thanks for your reply. The deployer has no such lookup/distearch file copy. So not really sure from where the SH are picking up the file after rolling restart.

Let me ask a different question-If I use the deployer to deploy an app (with distsearch.conf in it) so even if the file in etc/local gets corrupted, the precedence will always be given to the distsearch.conf in etc/apps/...

Will this work??

Thanks,

Neerav