Splunk Search

Bundle reload is in progress. Waiting for all peers to return the status.

phanichintha
Path Finder

My splunk environment is:
1 Search Head
1 Deployment Server (Master Node)
2 Indexers (Cluster)

I tried to implement retention policy to delete more than 365days old data in indexer, so i implement the sample script for one of the index(_internal).
So i first changed the indexes.conf under D:\Splunk\etc\master-apps_cluster\local in Master Node.
**Indexes.conf:
[_internal]
coldToFrozenDir = $SPLUNK_DB/_internal/frozendb
coldPath = $SPLUNK_DB/_internal/colddb
homePath = $SPLUNK_DB/_internal/db
thawedPath = $SPLUNK_DB/_internal/thaweddb

frozen time is 365 days

frozenTimePeriodInSecs = 31536000
maxHotIdleSecs = 3600
repFactor = auto*****

After same changed the indexes.conf under D:\Splunk\etc\slave-apps_cluster\local in indexer.
**Indexes.conf:
[_internal]
coldToFrozenDir = $SPLUNK_DB/_internal/frozendb
coldPath = $SPLUNK_DB/_internal/colddb
homePath = $SPLUNK_DB/_internal/db
thawedPath = $SPLUNK_DB/_internal/thaweddb

frozen time is 365 days

frozenTimePeriodInSecs = 31536000
maxHotIdleSecs = 3600
repFactor = auto*****

For doing changes, in master node GUI under settings > indexer cluster > push button.
After doing this from more than 3 hours its still (Bundle reload is in progress. Waiting for all peers to return the status.)
what is the time taken by doing this. Please any one help on priority.

0 Karma

codebuilder
Influencer

The reason you are having issues is that you manually edited indexes.conf on the indexers. This should never be done when you have clustered indexers controlled by a master node.

When you want to update/change indexes.conf on your cluster, you only modify the file on the master at /opt/splunk/etc/master-apps/_cluster/local/indexes.conf (linux). When you push the new bundle to the indexers from the master, the updated indexes.conf is written to /opt/splunk/etc/slave-apps/_cluster/local/indexes.conf on the peers (indexers).

The problem you are seeing is that the master cannot verify that the peers have the correct previous bundle because you modified them manually, and therefore will not push out the new changes.

Revert the manual changes you made on the peers/indexers and re-push the updated version from the master.

----
An upvote would be appreciated and Accept Solution if it helps!

codebuilder
Influencer

Also, the value you posted for repFactor is not valid..
You provided "repFactor = auto***"

The asterisks are not needed nor correct.
The value should be simply "repFactor = auto"

Worth noting as well, a two node indexer cluster is not good practice (though not the cause of your current issues). With only two nodes the cluster can never reach a quorum, so you're introducing a split-brain scenario.

----
An upvote would be appreciated and Accept Solution if it helps!
0 Karma
Get Updates on the Splunk Community!

Announcing Scheduled Export GA for Dashboard Studio

We're excited to announce the general availability of Scheduled Export for Dashboard Studio. Starting in ...

Extending Observability Content to Splunk Cloud

Watch Now!   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to leverage ...

More Control Over Your Monitoring Costs with Archived Metrics GA in US-AWS!

What if there was a way you could keep all the metrics data you need while saving on storage costs?This is now ...