Splunk Search

Bundle reload is in progress. Waiting for all peers to return the status.

phanichintha
Path Finder

My splunk environment is:
1 Search Head
1 Deployment Server (Master Node)
2 Indexers (Cluster)

I tried to implement retention policy to delete more than 365days old data in indexer, so i implement the sample script for one of the index(_internal).
So i first changed the indexes.conf under D:\Splunk\etc\master-apps_cluster\local in Master Node.
**Indexes.conf:
[_internal]
coldToFrozenDir = $SPLUNK_DB/_internal/frozendb
coldPath = $SPLUNK_DB/_internal/colddb
homePath = $SPLUNK_DB/_internal/db
thawedPath = $SPLUNK_DB/_internal/thaweddb

frozen time is 365 days

frozenTimePeriodInSecs = 31536000
maxHotIdleSecs = 3600
repFactor = auto*****

After same changed the indexes.conf under D:\Splunk\etc\slave-apps_cluster\local in indexer.
**Indexes.conf:
[_internal]
coldToFrozenDir = $SPLUNK_DB/_internal/frozendb
coldPath = $SPLUNK_DB/_internal/colddb
homePath = $SPLUNK_DB/_internal/db
thawedPath = $SPLUNK_DB/_internal/thaweddb

frozen time is 365 days

frozenTimePeriodInSecs = 31536000
maxHotIdleSecs = 3600
repFactor = auto*****

For doing changes, in master node GUI under settings > indexer cluster > push button.
After doing this from more than 3 hours its still (Bundle reload is in progress. Waiting for all peers to return the status.)
what is the time taken by doing this. Please any one help on priority.

0 Karma

codebuilder
Influencer

The reason you are having issues is that you manually edited indexes.conf on the indexers. This should never be done when you have clustered indexers controlled by a master node.

When you want to update/change indexes.conf on your cluster, you only modify the file on the master at /opt/splunk/etc/master-apps/_cluster/local/indexes.conf (linux). When you push the new bundle to the indexers from the master, the updated indexes.conf is written to /opt/splunk/etc/slave-apps/_cluster/local/indexes.conf on the peers (indexers).

The problem you are seeing is that the master cannot verify that the peers have the correct previous bundle because you modified them manually, and therefore will not push out the new changes.

Revert the manual changes you made on the peers/indexers and re-push the updated version from the master.

----
An upvote would be appreciated and Accept Solution if it helps!

codebuilder
Influencer

Also, the value you posted for repFactor is not valid..
You provided "repFactor = auto***"

The asterisks are not needed nor correct.
The value should be simply "repFactor = auto"

Worth noting as well, a two node indexer cluster is not good practice (though not the cause of your current issues). With only two nodes the cluster can never reach a quorum, so you're introducing a split-brain scenario.

----
An upvote would be appreciated and Accept Solution if it helps!
0 Karma
Get Updates on the Splunk Community!

Detecting Remote Code Executions With the Splunk Threat Research Team

WATCH NOWRemote code execution (RCE) vulnerabilities pose a significant risk to organizations. If exploited, ...

Enter the Splunk Community Dashboard Challenge for Your Chance to Win!

The Splunk Community Dashboard Challenge is underway! This is your chance to showcase your skills in creating ...

.conf24 | Session Scheduler is Live!!

.conf24 is happening June 11 - 14 in Las Vegas, and we are thrilled to announce that the conference catalog ...