Deployment Architecture

Why does a peer in an indexer cluster not have the "cluster_label" label?

bchau123
Engager

I have a clustered environment with many indexers, but for some reason one of the indexers are not playing nice with the rest of them.

Any time I restart the CM (Cluster Master), this indexer is always missing from the dashboard until I restart it. The CM is 6.4.5 as well as this troubled indexer. All the other indexers are on 6.3.2.

Upon further investigation of the cluster-config, I noticed that this trouble server is missing the cluster_label. All documentation says that this is supposed to be set on the master node, but I cant for the life of me figure out how to get it to propagate to this machine.

This problem server also seems to be having issues with synchronizing its indexes with the rest of the cluster. When this indexer is in the cluster, there is always some small portion of search and replication factors not being met (just a few buckets seem to be off).

Has anyone ran into this issue before? I cant seem to find any information on these symptoms.

0 Karma

chrishartsock
Path Finder

Has upgrading to 6.6 fixed this issue for you?

0 Karma

gjanders
SplunkTrust
SplunkTrust

I have tested on a 6.6 indexer and I no longer see this message post-restart of an indexer...

0 Karma

gjanders
SplunkTrust
SplunkTrust

I've had a case open with Splunk support on this topic since June 2016, I've just got the answer that Splunk 6.6 will fix this.

Adding the cluster label did not fix the issue in my environment.

0 Karma

adonio
Ultra Champion

hi bchau123,
please read here: http://docs.splunk.com/Documentation/Splunk/6.5.2/Indexer/Systemrequirements
specially this: "There are strict version compatibility requirements between cluster nodes"

0 Karma

bchau123
Engager

Hmm. This is a good read. Regarding my environment, Splunk Enterprise is installed on all machines and they are on the same network running the same OS. I do see the note further down regarding peer nodes need to run under the same version - down to the maintenance mode, but even further down, it says that mixed-version clusters are compatible from version 6.1+ on peers and 6.2 for the master. I have another environment that's running a similar splunk version disparity within the cluster (6.3.2 and 6.4.5 simultaneously - I am slowly migrating machines to 6.4.5 at the moment) that is not running into this problem.

0 Karma

adonio
Ultra Champion

try and put cluster peers in maintenance mode, then on the problem child, run this command:

splunk edit cluster-config -cluster_label <CLUSTER LABEL>

restart the peer and move out of maintenance mode

0 Karma

bchau123
Engager

Ive actually tried this before, but after setting the cluster_label and then listing the cluster-config, there is still no cluster_label in there - Even on restart. I've also ensured that permissions aren't an issue on anything in the /opt/splunk directory.

I really appreciate the suggestions and assistance.

0 Karma

adonio
Ultra Champion

What errors or warning do you get when you search index = _internal sourcetype = splunkd host = badIndexer log_level = warn OR log_level = error ?

0 Karma

bchau123
Engager

with this search, I get nothing. However, I dropped the "host=badIndexer" and I got TONS of "WARN ServerInfoHandler - Should not happen : Indexer cluster label should not be empty as it should default to CM's GUID" but that has already been established

0 Karma

adonio
Ultra Champion

when i put host=badindexer, i meant to replace "badIndexer" with your host that has 6.4.5 installed. can you share a message regarding the cluster label?
do you have cluster_label = <YourClusterLabel> on all Indexer Cluster Peers and Master?

0 Karma

bchau123
Engager

... yes that makes perfect sense. I replaced "badIndexer" with the problem child's hostname and I have tons of results saying "WARN ServerInfoHandler - Should not happen : Indexer cluster label should not be empty as it should default to CM's GUID."

The Master and all the other peers have the correct cluster_label in their cluster-config when I list them. It's just the problem server that seems to refuse to accept that setting.

0 Karma

adonio
Ultra Champion

is it single site or multisite?

0 Karma

bchau123
Engager

this is a single site cluster.

0 Karma

adonio
Ultra Champion

can you share server.conf of the bad indexer?

0 Karma

bchau123
Engager

The server.conf on the bad indexer is identical to the other indexers (except for serverName under [general] referencing itself).

Of what may be of interest, the [clustering] stanza lists only the master_uri and mode=slave. This is consistent between all indexers in the cluster.

0 Karma

adonio
Ultra Champion

do you have site = site<n> under default or under general?
do you use pass4symmkey?

0 Karma
Get Updates on the Splunk Community!

Take Your Breath Away with Splunk Risk-Based Alerting (RBA)

WATCH NOW!The Splunk Guide to Risk-Based Alerting is here to empower your SOC like never before. Join Haylee ...

SignalFlow: What? Why? How?

What is SignalFlow? Splunk Observability Cloud’s analytics engine, SignalFlow, opens up a world of in-depth ...

Federated Search for Amazon S3 | Key Use Cases to Streamline Compliance Workflows

Modern business operations are supported by data compliance. As regulations evolve, organizations must ...