And it seemed to never reduce down to 0 for the active_historical_search_count, but after 90 minutes, it seems to have come down to 0. We checked the currently-running searches and found some new searches running on the detention server after 1 hour. We have the following set in the server.conf:
decommission_force_finish_idle_time = 0
decommission_node_force_timeout = 300
decommission_search_jobs_wait_secs = 180
...so why is it taking 90 minutes to stop running savedsearches?
We did find some savedsearches that were running for long times and we fixed them, but should not all new searches be moved to another server once it is in manual detention? What can I do to fix this, so that my SHC can be upgraded?
When you have put SHC member manually into detention mode, it just wait that searches will finished.
On a search head that is in manual detention but not a part of a searchable rolling restart. These searches will run to completion.
On a search head that is a part of a rolling upgrade. During rolling upgrade of a search head cluster, you can put a single search head into manual detention and wait for the existing search jobs to run to completion before you shut down the search head.
I'm not sure if there is any way to force gracefully those sessions on SHC member? I usually just wait some time and after that cancel those jobs if needed.
I agree with you that on detention mode it shouldn't accept any new queries from schedule or users anymore. But maybe there is error in documentation and it means that it don't accept any new sessions from user to this node?