Splunk unable to read files

[ankithreddy777]

Hi,
Splunk UF is setup to read files from particular directory. It reads files normally for few minutes, but suddenly it throws an error

WARN FilesystemChangeWatcher - error reading directory "xxxx/xxxx/xxx": Permission denied

Again when we reinstall splunk it reads normally and stops after few minutes showing permission denied.

May I know, Is there any reason that it suddenly shows permission denied error after reading half of the file.

Is there any possibility that when a file is written by some other process , It blocks other process to read the file?

[RohitMu]

have you checked the permission for that directory and if the user running splunk had read permissions for that directory?

[ankithreddy777]

yes there are sufficient permissions. We could sudo to splunk user and open files to read

[hortonew]

This a UF on windows or linux? What's OS / splunk UF version?

[ankithreddy777]

UF on linux. Redhat 7, splunk UF 6.5.7

[hortonew]

Are the ones that are failing in /var/log ? Can you confirm if you've set permissions there manually or have configured a custom umask on them that will allow "other" or a group the splunk user is part of to read/execute on directories and read files? Also can you confirm that you're not manually setting permissions there when you reinstall splunk, it "just works" after the reinstall.

[ankithreddy777]

Splunk user is the part of the same group the file is part of
Ex:

drwxrwx--- user abc wer.log

Splunk user is part of abc group who has read permissions.
I haven't set any Manuel permissions

[hortonew]

Can you change to the splunk user (su - splunk) when the error occurs and less <file.log> without issue?

[ankithreddy777]

Yes, I can sudo to splunk user and view the file.