Deployment Architecture

Distsearch.conf is not updated on SH after adding new peers

Path Finder

Hey,
I noticed a problem on my clustered environment, when the SH could not search over 2 new peers I’ve added to the cluster earlier.

When trying to search over the new peers’ ‘_internal’ logs, no logs where shown. But when searching for the same on the cluster master, I found the events.

Note that the new peers were not marked as quarantined, but they did appear in the Disturbuted Search Peers list.

I noticed that the monitoring console did not show them on the Resource Usage section, which using the dmc lookup, so I found out a solution - I had to manually add the peers to the ‘distsearch.conf’ on SH (SPLUNK_HOME/etc/system/local/distsearch.conf)

I wonder why the peers where not in the file already, as the others were in it, and I never had to change it before.

Is it a bug? Would I have to do it each time adding a new peer or is there a better way to handle it?

Thanks!

0 Karma
1 Solution

Influencer

Another thing:

Did you make sure that your Monitoring Cosole (I just assume that your SH is your MC) is applying the correct role for the new indexers? and make sure you apply the new settings in global settings of your MC. This might affect your search peer config.

Also you want to delete the manually updated distsearch.conf, since this could cause duplicate events, the SH is not aware that these new indexers are clustered indexes if you add this manually.

View solution in original post

0 Karma

Influencer

Another thing:

Did you make sure that your Monitoring Cosole (I just assume that your SH is your MC) is applying the correct role for the new indexers? and make sure you apply the new settings in global settings of your MC. This might affect your search peer config.

Also you want to delete the manually updated distsearch.conf, since this could cause duplicate events, the SH is not aware that these new indexers are clustered indexes if you add this manually.

View solution in original post

0 Karma

Influencer

Did you try the above ?

0 Karma

Path Finder

Well, I found out that the monitoring console setting page was the solution! The roles where correct, all I had to do is press Apply Settings. Weird, but it worked. Thanks!

0 Karma

Influencer
0 Karma

Path Finder

Well, this is a bit different situation. I have a single search head, not a cluster, and it fails to add new Peers (indexers) to its dmc group.

0 Karma

Influencer

Ah ok sry, I did got this confused because its saying "cluster" in the question.

Did you add the SH as SH for the index cluster?
https://docs.splunk.com/Documentation/Splunk/7.2.3/Indexer/Enablethesearchhead

0 Karma

Path Finder

Yes, and it shows up in the Search Heads section on the Cluster Master “indexer clustering” page

0 Karma

Influencer

Do you see any errors in splunkd.log of your SH? or Indexer peers that can´t be searched?

0 Karma

Path Finder

No I have not noticed anything, it was like this for weeks. Was it a good solution to add them manually? Shouldn’t it update automatically?

0 Karma

Influencer

it should be added automatically if the SH is configured as a Index Cluster Searchhead yes.

0 Karma