Why would Splunk NOT obey "dispatch.ttl" and delete results/artifacts early?

[woodcock]

We have a not-at-all overloaded ES search head with a separate volume for dispatch with plenty of room that gives us 500MB warnings. We also have a few weekly-scheduled searches which bring back 100ish rows of results with dozensish fields with default values of "2p" for "dispatch.ttl" but the results are always gone after 2 days. We are on 7.3.latest.

We have tried setting it to 2 weeks worth of seconds and that did not work. What could be causing this? What logs should I look at/for?

[matthewhasty · Accepted solution]

I don't think it has to do with the amount of space in your dispatch directory. If it was completely full it should not delete the jobs, instead it should not allow any more searches to be dispatched. Do these searches have any addition actions such as e-mail, etc? The ttl for those actions may be overwriting. Alert actions like e-mail have a live time of 24 hours, which when taken with the default of 2x this value, would put it at 2 days, which is exactly what you are seeing.

alert_actions.conf is where this would be modified.