We've recently moved our production search heads to a search head cluster, since last week (6.2.6?) I have noticed that any ad-hoc jobs (via REST API or WEB UI) are not expiring and quickly stack up.
I've checked the limits.conf and savedsearches.conf, and have confirmed that the ttl's are set to 600 seconds or less.
This only happens in a clustered environment. We have dev servers running the exact same searches without issue.
In the job inspector info below, I can see that the job was created yesterday. It has completed successfully and has TTLs of 600 seconds, so why is it still there?? The expiration time just updates to now whenever I refresh the jobs list.
Is there some config specific to SHC that sets the TTL for completed jobs?