I keep coming across the issue where certain scheduled searches that user may set up will keep artifacts in the dispatch directory for 24 hours, even though the retention is set to only a few minutes.
These searches run every minute, sometimes for alerts, sometimes to populate a summary or dashboard, so it's not really search specific.
I believe this has to do with how long to keep an artifact if there is some transient error in the search, and I believe this is why these are being kept beyond the set retention time. I can not find this setting however.
I've been asking this question for years and no one has come forward. After a lot of testing there are several settings -- in different config files -- which will affect how long the search artifact will be stored. I figured some of these out by documenting the behavior and checking all the default settings in Splunk.
Here are some of the settings I have played with and use:
ttl = 120
ttl = 2p
ttl = 120
dispatch.ttl = 2p
alert.expires = 2m
alert.severity = 2
ttl = 300
default_save_ttl = 172800
# if user clicks to save search it is stored for 1 week
remote_ttl = 300