If permissions get changed on a log file that is being monitored by splunk and becomes unreadable, will Splunk give up permanently on rechecking the readability of the log without an agent restart? Just wondering what the testing logic is and whether the agent will resume collecting the logs once the log file permissions are returned to readable?
The behavior changed in 4.3.3, before the Tailing processor was trying 10 times to read a file then gave up (if permission were restrictives or the file locked). So the only way was to restart the forwarder to redetect the file (or use the REST endpoint to reload the tailing processor)
Since late 4.3.3 and 5.0.*, the Tailing processor retry after in interval of time, this interval increase with each failure to maximum of 34 minutes.
When tailing files, consecutive failures for accessing or handling the same pathname will result in a doubling interval up to 0.5s * 2^(12), or 1 * 2^(11) seconds or 2048 seconds, or 34 minutes and 8 seconds. If more errors are encountered, the timer will remain at 34 minutes and 8 seconds. (SPL-50995)