The splunkd.log file on a search head cluster member shows 450K of this type of error:
05-01-2016 18:42:15.010 -0400 WARN SHPMaster - did not schedule removal for peer='E984FB44-0D11-4B0B-ABBF-95E9BAE41658', err='SHPMaster::scheduleRemoveArtifactFromPeer_locked: aid=scheduler__admin__search__RMD51fa2da2a876e07a8_at_1461834240_5_E984FB44-0D11-4B0B-ABBF-95E9BAE41658 peer=E984FB44-0D11-4B0B-ABBF-95E9BAE41658 is pending some change, status='PendingDiscard''
Clarification on this error messages should be useful.
max_searches_per_process = <int>
* On UNIX, specifies the maximum number of searches that each search process
can run before exiting.
* After a search completes, the search process can wait for another search to
start and the search process can be reused.
* When set to “0” or “1”: The process is never reused.
* When set to a negative value: There is no limit to the number of searches
that a process can run.
* Has no effect on Windows if search_process_mode is not "auto”.
* Default: 500
If you find the "/services/shcluster/member/artifacts//discard endpoint failing due to network/REST issues on splunkd_access.log on the members this can happen.
Other possibility is lot of messages between captain/members leading to delay in messages..
For the 1st case, recommendation is to transfer captaincy to a different node.. (6.2.3 does not have transfer captaincy command) So manually transfer captaincy by bringing down the current captain will fix the current flow of WARN messages
if you don't find any network issues/REST call failures it could be the delay in jobs and increasing the "executor_workers = 20" (default 10) in stanza shclustering in server.conf might help..