splunk offline --enforce-counts looks stuck after 3 days of the decommission on first indexer of a multi site cluster

[veryfoot]

Hi all,

I'm actually have to decomission 6 indexers on a 9/9 multi site cluster of indexers.

The command passed :

splunk offline --enforce-counts

3 days have passed, and im still having a large amount of buckets for the offlined indexer. Buckets dont reduce... or a very little amount.

The Indexer is still in "Decomissionning" status in the Cluster master (setting/indexer clustering)

The RP/SF is KO.

There is no more active tasks (all complete around 12 000 tasks performed and OK) exept for 4 tasks who are waiting the RF/SF back to OK. (pending)

All the indexers of both site are communicating well ones with others.

Does anybody have all ready encounter this problem ?

I have checked errors messages (splunkd.log) in CM / Decomissionned indexer / and other indexers and I dont find any revealant messages or errors.

Is it safe to launch a rolling restart ?

Or to shoud I restart splunkd on the decommissionned indexer?

Thanks for any help

[richgalloway · Accepted solution]

Do not restart the decommissioned indexer.

If the indexer stopped running then it has finished its work and the server can be retired.  Consider restarting the CM to force it to rebuild the bucket table.

[veryfoot]

Thanks for your return,

You are right. The decomissionned indexer is now on state "Graceful shutdown" and buckets count is 0.

Took 2.5 days to decomission 20 To of datas. 

But SF / RF is still not green.

3 SF tasks are still in pending, i tried to resync thems but no change. 

Should I now do a rolling restart after removed my decomissionned indexer in order to get back my SF / RP ? 

Or simply restart my CM splunk deamon ?

An other intorragation, is it normal to only have default DataModels visible (and not all my Datamodels) from CM (Settings/DataModels)  ?

• Many thanks 

[richgalloway]

Restart the CM first.

[veryfoot]

》An other intorragation, is it normal to only have default DataModels visible (and not all my Datamodels) from CM (Settings/DataModels)  ?

My DM are ok.... sorry for that