After changing data retention, why is indexes.conf not working on one indexer?

[kiran331]

Hi

I have changed the data retention and pushed the bundle from the cluster master. In 2 indexers, the data got deleted but in one indexer, it's still the same. The indexes.conf in the slave-apps is same on all 3 indexers. What changes do I have to make so that the indexer will remove data from the cold buckets of one index?

Splunk version - 6.4.2

[dxu_splunk]

try using btool to figure out which file is providing which configuration.

bin/splunk btool indexes list INDEX_IN_QUESTION --debug

[kiran331]

it shows the correct file and its same on other 2 indexers, the issue is only for that one index.

[vasanthmss]

hope the below debugging steps will helps you,

1. run this on cluster master, /opt/splunk/bin/splunk show cluster-bundle-status and validate the bundle status. all indexers should have the same bundle id and time.
2. keep the cluster master in maintenance mode /opt/splunk/bin/splunk enable maintenance-mode and restart all the indexers. hopefully this will fix the issue. then disable the maintenance mode /opt/splunk/bin/splunk disable maintenance-mode

Cheers!!

[kiran331]

Hi Vasanthmss, thanks for your response, I tried restarting all indexers, one indexer is still same.

[vasanthmss]

check the other indexers, etc/system/* and etc/apps/*

Seems like other stanza overriding your bundle priority.