Why are universal forwarder internal logs not getting rotated due to permission issues (Access is denied) in Windows?

[koshyk]

We have deployed universal forwarders on Windows and are running as "local system" (admin). This is installed in C:\Program Files\SplunkUniversalForwarder. When we checked into the splunkd.log details, none of the logs are getting rotated due to permission issues:

WARN Logger - Error unlinking "C:\Program Files\SplunkUniversalForwarder\var\log\splunk\metrics.log.1":  Access is denied
WARN Logger - Error renaming "C:\Program Files\SplunkUniversalForwarder\var\log\splunk\metrics.log"  to "C:\Program Files\SplunkUniversalForwarder\var\log\splunk\metrics.log.1":  Access is denied

As an admin, I can read/write into the same folder. Splunkd can write the log files Ok as the data and size is growing in each of the files. Any reason why access is denied when it tries to rename/unlink?

[gavsdavs_GR]

This one is confusing, it's happening on a number of machines here.

The sequence is (or should be):
metrics.log.5 gets deleted
metrics.log.4 is renamed to metrics.log.5
metrics.log.3 is renamed to metrics.log.4
metrics.log.2 is renamed to metrics.log.3
metrics.log.1 is renamed to metrics.log.2
metrics.log is renamed to metrics.log.1
a new metrics.log is created.

We are seeing all permissions removed on metrics.log.5
(i.e. an adminstrator has no permissions on the file to even inspect permissions)

This prevents the above sequence from occurring and our metrics.log files are getting larger and larger.

We do not understand what might be interfering with the permissions of the metrics.log.5 file, since all the other files are acciessible, manageable.

I am pretty sure we don't have people looking at metrics.log.5 with a notepad. It's also happening on a number of machines.

Can a splunk person comment on the sequence of actions taken by the UF when rolling out metrics.log.5 ?

We can't tell if something we have in place is occasionally interfering with the removal of it.

[javiergn]

That usually happens when you have a lock on those files somehow.
I've seen it when using tail or notepad.
Make sure nothing is reading from your metrics.log as that's the one that can't be renamed.

[koshyk]

nothing is reading the file other than Splunk UniversalForwader itself trying to send to Indexer

[javiergn]

Can you try using Procexp to double check that?
http://stackoverflow.com/questions/320128/releasing-windows-file-share-locks
http://www.howtogeek.com/128680/how-to-delete-move-or-rename-locked-files-in-windows/

If nothing is locking it according to Procexp, try restarting Splunk and it that works then it probably means Splunk was locking those files and that's not great. I would raise a support request but I guess you might be asked to replicate the problem and that might not be easy.

[koshyk]

thank you for your assistance. I will hopefully raise a support request