Distsearch.conf is not updated on SH after adding new peers

[omerl]

Hey,
I noticed a problem on my clustered environment, when the SH could not search over 2 new peers I’ve added to the cluster earlier.

When trying to search over the new peers’ ‘_internal’ logs, no logs where shown. But when searching for the same on the cluster master, I found the events.

Note that the new peers were not marked as quarantined, but they did appear in the Disturbuted Search Peers list.

I noticed that the monitoring console did not show them on the Resource Usage section, which using the dmc lookup, so I found out a solution - I had to manually add the peers to the ‘distsearch.conf’ on SH (SPLUNK_HOME/etc/system/local/distsearch.conf)

I wonder why the peers where not in the file already, as the others were in it, and I never had to change it before.

Is it a bug? Would I have to do it each time adding a new peer or is there a better way to handle it?

Thanks!

[dkeck · Accepted solution]

Another thing:

Did you make sure that your Monitoring Cosole (I just assume that your SH is your MC) is applying the correct role for the new indexers? and make sure you apply the new settings in global settings of your MC. This might affect your search peer config.

Also you want to delete the manually updated distsearch.conf, since this could cause duplicate events, the SH is not aware that these new indexers are clustered indexes if you add this manually.

[dkeck]

Did you try the above ?

[omerl]

Well, I found out that the monitoring console setting page was the solution! The roles where correct, all I had to do is press Apply Settings. Weird, but it worked. Thanks!

[dkeck]

Hi,

did you see the post from jkat54? https://answers.splunk.com/answers/405041/how-to-distribute-distributed-search-configuration-1.html

[omerl]

Well, this is a bit different situation. I have a single search head, not a cluster, and it fails to add new Peers (indexers) to its dmc group.

[dkeck]

Ah ok sry, I did got this confused because its saying "cluster" in the question.

Did you add the SH as SH for the index cluster?
https://docs.splunk.com/Documentation/Splunk/7.2.3/Indexer/Enablethesearchhead

[omerl]

Yes, and it shows up in the Search Heads section on the Cluster Master “indexer clustering” page

[dkeck]

Do you see any errors in splunkd.log of your SH? or Indexer peers that can´t be searched?

[omerl]

No I have not noticed anything, it was like this for weeks. Was it a good solution to add them manually? Shouldn’t it update automatically?

[dkeck]

it should be added automatically if the SH is configured as a Index Cluster Searchhead yes.