Deployment Architecture

Distsearch.conf is not updated on SH after adding new peers

omerl
Path Finder

Hey,
I noticed a problem on my clustered environment, when the SH could not search over 2 new peers I’ve added to the cluster earlier.

When trying to search over the new peers’ ‘_internal’ logs, no logs where shown. But when searching for the same on the cluster master, I found the events.

Note that the new peers were not marked as quarantined, but they did appear in the Disturbuted Search Peers list.

I noticed that the monitoring console did not show them on the Resource Usage section, which using the dmc lookup, so I found out a solution - I had to manually add the peers to the ‘distsearch.conf’ on SH (SPLUNK_HOME/etc/system/local/distsearch.conf)

I wonder why the peers where not in the file already, as the others were in it, and I never had to change it before.

Is it a bug? Would I have to do it each time adding a new peer or is there a better way to handle it?

Thanks!

0 Karma
1 Solution

dkeck
Influencer

Another thing:

Did you make sure that your Monitoring Cosole (I just assume that your SH is your MC) is applying the correct role for the new indexers? and make sure you apply the new settings in global settings of your MC. This might affect your search peer config.

Also you want to delete the manually updated distsearch.conf, since this could cause duplicate events, the SH is not aware that these new indexers are clustered indexes if you add this manually.

View solution in original post

0 Karma

dkeck
Influencer

Another thing:

Did you make sure that your Monitoring Cosole (I just assume that your SH is your MC) is applying the correct role for the new indexers? and make sure you apply the new settings in global settings of your MC. This might affect your search peer config.

Also you want to delete the manually updated distsearch.conf, since this could cause duplicate events, the SH is not aware that these new indexers are clustered indexes if you add this manually.

0 Karma

dkeck
Influencer

Did you try the above ?

0 Karma

omerl
Path Finder

Well, I found out that the monitoring console setting page was the solution! The roles where correct, all I had to do is press Apply Settings. Weird, but it worked. Thanks!

0 Karma

dkeck
Influencer
0 Karma

omerl
Path Finder

Well, this is a bit different situation. I have a single search head, not a cluster, and it fails to add new Peers (indexers) to its dmc group.

0 Karma

dkeck
Influencer

Ah ok sry, I did got this confused because its saying "cluster" in the question.

Did you add the SH as SH for the index cluster?
https://docs.splunk.com/Documentation/Splunk/7.2.3/Indexer/Enablethesearchhead

0 Karma

omerl
Path Finder

Yes, and it shows up in the Search Heads section on the Cluster Master “indexer clustering” page

0 Karma

dkeck
Influencer

Do you see any errors in splunkd.log of your SH? or Indexer peers that can´t be searched?

0 Karma

omerl
Path Finder

No I have not noticed anything, it was like this for weeks. Was it a good solution to add them manually? Shouldn’t it update automatically?

0 Karma

dkeck
Influencer

it should be added automatically if the SH is configured as a Index Cluster Searchhead yes.

0 Karma
Get Updates on the Splunk Community!

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...