Here's a possible situation. Your indexers appear to be under stress and not performing fast enough for the inputs. Then due to the busy indexers, your forwarders also unable to send data and put logs like " TailReader - Could not send data to output queue " and at the same time your application rolls log data over to another directory where UF is not monitoring files.
Here's the suggestion - please share it with your application team.
i) Your application team has to change the rotation mechanism to like below;
> How to rotate:
Rotate files as before but keep the rotated file in the same directory as "/logs" so that splunk forwarder keeps reading the rest regardless of the name change.
> When to move rotated files to the other directory.
Use below command and retrieve "sptr=", this value has to be the same as the file size of "ls -l" or stat . Then you can move it to the other safely.
"command sample" > ./splunk cmd btprobe -d ../var/lib/splunk/fishbucket/splunk_private_db --file /logs/abcACSLog_20180517.txt
"command resutl" > key=0x340aa4a716415f07 scrc=0xac3413d846a972c9 sptr=14822 fcrc=0xc7cee9b9b8bf07ca flen=0 mdtm=1429060083 wrtm=1429060148
Or there are some other ways like, rotate file first and then move the previously rotated file later. But this is not safe way to move.
ii) Your forwarder input has to be changed to keep reading the rotated file.
---- input config
[monitor://logs/abcACSLog.txt] ===> [monitor://logs/abcACSLog*.txt] to cover the rotated files.
Or just add one more monitor to cover the backup directory.
