I'm encountering an issue where, after changing or creating a new script-based app that runs periodically (e.g. once per hour), the forwarders check-in, deploy the app, then all, perhaps 1000's, restart and execute the script at effectively the same time. Among other issues, this causes an unwanted spike in traffic.
To fix this, either the forwarders need to check in after longer, hopefully random times, or the forwarders need to not all restart at the same time. I'd like to not have to add a random wait at the start of each script if I can help it.
Is there a way to have the forwarders restart after a random delay generated by each forwarder?
I think this should be a feature of the Splunk forwarders.