Splunk UF is setup to read files from particular directory. It reads files normally for few minutes, but suddenly it throws an error
WARN FilesystemChangeWatcher - error reading directory "xxxx/xxxx/xxx": Permission denied
Again when we reinstall splunk it reads normally and stops after few minutes showing permission denied.
May I know, Is there any reason that it suddenly shows permission denied error after reading half of the file.
Is there any possibility that when a file is written by some other process , It blocks other process to read the file?
Are the ones that are failing in /var/log ? Can you confirm if you've set permissions there manually or have configured a custom umask on them that will allow "other" or a group the splunk user is part of to read/execute on directories and read files? Also can you confirm that you're not manually setting permissions there when you reinstall splunk, it "just works" after the reinstall.
Splunk user is the part of the same group the file is part of
drwxrwx--- user abc wer.log
Splunk user is part of abc group who has read permissions.
I haven't set any Manuel permissions