topic Re: Scheduled searches getting skipped due to rolling restart in Splunk Enterprise

Why are scheduled searches getting skipped due to rolling restart?

shadysplunker — Thu, 04 May 2023 12:48:28 GMT

We have an issue where all the scheduled searches are getting skipped whenever rolling restart is in progress.

Also, since few weeks, we have observed that the cluster master automatically initiates a rolling restart of the indexers, twice in a week. It takes about 24 hours to restart all the 24 indexers in a cluster, which impacts our business too.

Has anyone ever incurred this situation before?

Re: Scheduled searches getting skipped due to rolling restart

PickleRick — Thu, 04 May 2023 10:03:37 GMT

Master should _not_ initiate restart out of thin air. There must be something that triggers it.

Also see https://docs.splunk.com/Documentation/Splunk/latest/Indexer/Userollingrestart#How_searchable_rolling_restart_works

The timeouts and wait times can impact whether your searches get skipped.

Re: Scheduled searches getting skipped due to rolling restart

shadysplunker — Thu, 04 May 2023 10:24:21 GMT

Thanks for pointing me out to a proper direction!

Here's our config from [clustering] stanza:

[clustering]
mode = master
multisite = true
available_sites = site1, site2
site_replication_factor = origin:1, site1:1, site2:1, total:3
site_search_factor = origin:1, site1:1, site2:1, total:2
cluster_label = cluster1
maintenance_mode = false
max_peers_to_download_bundle = 10
service_interval = 10
heartbeat_timeout = 1800
cxn_timeout = 300
send_timeout = 300
rcv_timeout = 300
max_peer_build_load = 5
rolling_restart = searchable
restart_timeout = 500
decommission_force_timeout = 900
restart_inactivity_timeout = 1500
rebalance_threshold = 0.96
max_auto_service_interval = 250

I suspect few things here like "rolling_restart" should be "searchble_force" and max_peers_to_download_bundle would be more than 10? considering we have 24 indexers.

I will go through all these parameters and understand it in detail.

Do you suspect anything unusual in the configuration here? It would be much helpful! Thanks!