scheduling task Zombie scheduling tasks or residual tasks in the scheduler are stuck

[connoe]

I found that the audit data volume was abnormal, with a large number of interrupted and blocked logs. Based on search_id, I checked the corresponding tasks and found that some tasks had been disabled and some had been deleted, but audit showed that they were still being scheduled and were interrupted and blocked very quickly. splunk 7.2.4

[connoe]

25/10/23 13:49:14.681
Audit:[timestamp=10-23-2025 13:49:14.681, user=splunk-system-user, action=search, info=terminate, search_id='scheduler__admin_Y3JjYl9zb2M__RMD521469da3b4d98e5c_at_1608232140_66522_DFDF466D-7806-45A7-8AF8-668D0FD6DA89'][n/a]
25/10/23 13:49:14.681
Audit:[timestamp=10-23-2025 13:49:14.681, user=splunk-system-user, action=search, info=cancel, search_id='scheduler__admin_Y3JjYl9zb2M__RMD521469da3b4d98e5c_at_1608232140_66522_DFDF466D-7806-45A7-8AF8-668D0FD6DA89'][n/a]
25/10/23 13:49:14.681
Audit:[timestamp=10-23-2025 13:49:14.681, user=splunk-system-user, action=search, info=terminate, search_id='scheduler__manager_Y3JjYl9zb2M__RMD5e4bf144359145290_at_1736159640_85246_2F9B3DBA-D627-42DA-8463-E30EA1FE1D2F'][n/a]

I have three search headers. Only one of them, the audit, is filled with such logs, and the data volume is extremely large. As a result, I now have tens of millions of audit logs every day. I followed some rules and found that it was once per minute, but in audi, it can generate around 7,000 such logs per minute

[livehybrid]

Hi @connoe 

Firstly, Splunk 7.2.4 went out of support in April 2021 so you should certainly look into upgrading - any bugs relating to this issue could well be resolved in future versions, not to mention other performance/security/features since then too.

In the meantime if you're able to share the specific logs you are referring to then we might be able to help dig into this issue.

🌟 Did this answer help you? If so, please consider:

• Adding karma to show it was useful
• Marking it as the solution if it resolved your issue
• Commenting if you need any clarification

Your feedback encourages the volunteers in this community to continue contributing