I figured out the issue. It was a permissions issue. I needed to put splunkfwd on the appropriate access lists. I gave splunkfwd read access to /var/log/audit/audit.log and execute access to /var/...
See more...
I figured out the issue. It was a permissions issue. I needed to put splunkfwd on the appropriate access lists. I gave splunkfwd read access to /var/log/audit/audit.log and execute access to /var/log/audit. Now splunkfwd can execute the script either manually from command line or as a scheduled scripted input run by Splunk UF. In both cases, the script runs without error whether there is a pre-existing checkpoint in place or not. I understand that Splunk UF has the CAP_DAC_READ_SEARCH capability which allows it to read files it normally wouldn't have access to. What I don't understand is why that capability worked fine when I asked it to generate the initial checkpoint, but then suddenly stopped working the moment that I asked it to use a pre-existing checkpoint. Is it possible that the CAP_DAC_READ_SEARCH capability doesn't extend to reading the inode properties of each file? If that were the case, it would explain why the initial ausearch went fine (when inode doesn't matter because ausearch is just ingesting all of the audit.log files regardless of inode), but then when ausearch needs to look for the specific audit.log file that matches the inode listed in the checkpoint file, it can't do so. Thank you to @PickleRick and @isoutamo for your suggestions and assistance. I couldn't have done it without you both.