Deployment Architecture

splunk offline --enforce-counts looks stuck after 3 days of the decommission on first indexer of a multi site cluster

veryfoot
Path Finder

Hi all,

I'm actually have to decomission 6 indexers on a 9/9 multi site cluster of indexers.

The command passed :

splunk offline --enforce-counts

3 days have passed, and im still having a large amount of buckets for the offlined indexer. Buckets dont reduce... or a very little amount.

The Indexer is still in "Decomissionning" status in the Cluster master (setting/indexer clustering)

The RP/SF is KO.

There is no more active tasks (all complete around 12 000 tasks performed and OK) exept for 4 tasks who are waiting the RF/SF back to OK. (pending)

All the indexers of both site are communicating well ones with others.

Does anybody have all ready encounter this problem ?

I have checked errors messages (splunkd.log) in CM / Decomissionned indexer / and other indexers and I dont find any revealant messages or errors.

Is it safe to launch a rolling restart ?

Or to shoud I restart splunkd on the decommissionned indexer?

Thanks for any help

Labels (1)
Tags (1)
0 Karma
1 Solution

richgalloway
SplunkTrust
SplunkTrust

Do not restart the decommissioned indexer.

If the indexer stopped running then it has finished its work and the server can be retired.  Consider restarting the CM to force it to rebuild the bucket table.

---
If this reply helps you, Karma would be appreciated.

View solution in original post

richgalloway
SplunkTrust
SplunkTrust

Do not restart the decommissioned indexer.

If the indexer stopped running then it has finished its work and the server can be retired.  Consider restarting the CM to force it to rebuild the bucket table.

---
If this reply helps you, Karma would be appreciated.

veryfoot
Path Finder

Thanks for your return,

You are right. The decomissionned indexer is now on state "Graceful shutdown" and buckets count is 0.

Took 2.5 days to decomission 20 To of datas. 

But SF / RF is still not green.

3 SF tasks are still in pending, i tried to resync thems but no change. 

Should I now do a rolling restart after removed my decomissionned indexer in order to get back my SF / RP ? 

Or simply restart my CM splunk deamon ?

An other intorragation, is it normal to only have default DataModels visible (and not all my Datamodels) from CM (Settings/DataModels)  ?

  • Many thanks 

 

0 Karma

richgalloway
SplunkTrust
SplunkTrust

Restart the CM first.

---
If this reply helps you, Karma would be appreciated.
0 Karma

veryfoot
Path Finder

An other intorragation, is it normal to only have default DataModels visible (and not all my Datamodels) from CM (Settings/DataModels)  ?

My DM are ok.... sorry for that

0 Karma
Get Updates on the Splunk Community!

Introducing the 2024 SplunkTrust!

Hello, Splunk Community! We are beyond thrilled to announce our newest group of SplunkTrust members!  The ...

Introducing the 2024 Splunk MVPs!

We are excited to announce the 2024 cohort of the Splunk MVP program. Splunk MVPs are passionate members of ...

Splunk Custom Visualizations App End of Life

The Splunk Custom Visualizations apps End of Life for SimpleXML will reach end of support on Dec 21, 2024, ...