I have a custom application that does incremental loads from an external resource. I maintain a file with the latest timestamps so that each run I can read the file and pull the new data. My struggle is with how do I perform application upgrades without rolling these files back to their initial value. Is there a configuration that can be set to ignore file types or directories during a custom application upgrade?
The "modular input way", which I also would suggest for other types of inputs, is to use the "checkpoint_dir" configuration. When A modular input is created, it is assigned a special checkpoint directory to store such files that keep checkpoints. Each file contains whatever information is needed for the input (either a single timestamp, or perhaps a json object with a more complex status). The location for modular input is /opt/splunk/var/lib/splunk/modinputs/$INPUTNAME. This will do a few things for you:
Remove dependency on default and local configs. You could remove the entire App, reinstall, and as long as the script looks in the checkpoint dir, you are fine.
A consistent place to put the files, and won't (read shouldn't) break across Splunk upgrades.
As always, you can contact me directly via email, IRC (#splunk on EfNet), or just shouting at the screen (ok, maybe not that one).