topic Re: Distsearch.conf is not updated on SH after adding new peers in Deployment Architecture

Distsearch.conf is not updated on SH after adding new peers

omerl — Wed, 09 Jan 2019 06:58:27 GMT

Hey,
I noticed a problem on my clustered environment, when the SH could not search over 2 new peers I’ve added to the cluster earlier.

When trying to search over the new peers’ ‘_internal’ logs, no logs where shown. But when searching for the same on the cluster master, I found the events.

Note that the new peers were not marked as quarantined, but they did appear in the Disturbuted Search Peers list.

I noticed that the monitoring console did not show them on the Resource Usage section, which using the dmc lookup, so I found out a solution - I had to manually add the peers to the ‘distsearch.conf’ on SH (SPLUNK_HOME/etc/system/local/distsearch.conf)

I wonder why the peers where not in the file already, as the others were in it, and I never had to change it before.

Is it a bug? Would I have to do it each time adding a new peer or is there a better way to handle it?

Thanks!

Re: Distsearch.conf is not updated on SH after adding new peers

dkeck — Wed, 09 Jan 2019 07:09:02 GMT

Hi,

did you see the post from jkat54? https://answers.splunk.com/answers/405041/how-to-distribute-distributed-search-configuration-1.html

Re: Distsearch.conf is not updated on SH after adding new peers

omerl — Wed, 09 Jan 2019 07:28:42 GMT

Well, this is a bit different situation. I have a single search head, not a cluster, and it fails to add new Peers (indexers) to its dmc group.

Re: Distsearch.conf is not updated on SH after adding new peers

dkeck — Wed, 09 Jan 2019 07:34:27 GMT

Ah ok sry, I did got this confused because its saying "cluster" in the question.

Did you add the SH as SH for the index cluster?
https://docs.splunk.com/Documentation/Splunk/7.2.3/Indexer/Enablethesearchhead

Re: Distsearch.conf is not updated on SH after adding new peers

omerl — Wed, 09 Jan 2019 07:39:11 GMT

Yes, and it shows up in the Search Heads section on the Cluster Master “indexer clustering” page

Re: Distsearch.conf is not updated on SH after adding new peers

dkeck — Wed, 09 Jan 2019 07:47:25 GMT

Do you see any errors in splunkd.log of your SH? or Indexer peers that can´t be searched?

Re: Distsearch.conf is not updated on SH after adding new peers

omerl — Wed, 09 Jan 2019 08:10:29 GMT

No I have not noticed anything, it was like this for weeks. Was it a good solution to add them manually? Shouldn’t it update automatically?

Re: Distsearch.conf is not updated on SH after adding new peers

dkeck — Wed, 09 Jan 2019 08:12:31 GMT

it should be added automatically if the SH is configured as a Index Cluster Searchhead yes.

Re: Distsearch.conf is not updated on SH after adding new peers

dkeck — Wed, 09 Jan 2019 08:37:18 GMT

Another thing:

Did you make sure that your Monitoring Cosole (I just assume that your SH is your MC) is applying the correct role for the new indexers? and make sure you apply the new settings in global settings of your MC. This might affect your search peer config.

Also you want to delete the manually updated distsearch.conf, since this could cause duplicate events, the SH is not aware that these new indexers are clustered indexes if you add this manually.

Re: Distsearch.conf is not updated on SH after adding new peers

dkeck — Thu, 10 Jan 2019 14:08:06 GMT

Did you try the above ?

Re: Distsearch.conf is not updated on SH after adding new peers

omerl — Sun, 13 Jan 2019 19:45:41 GMT

Well, I found out that the monitoring console setting page was the solution! The roles where correct, all I had to do is press Apply Settings. Weird, but it worked. Thanks!