Getting Data In

Why are universal forwarder internal logs not getting rotated due to permission issues (Access is denied) in Windows?

Super Champion

We have deployed universal forwarders on Windows and are running as "local system" (admin). This is installed in C:\Program Files\SplunkUniversalForwarder. When we checked into the splunkd.log details, none of the logs are getting rotated due to permission issues:

WARN Logger - Error unlinking "C:\Program Files\SplunkUniversalForwarder\var\log\splunk\metrics.log.1":  Access is denied
WARN Logger - Error renaming "C:\Program Files\SplunkUniversalForwarder\var\log\splunk\metrics.log"  to "C:\Program Files\SplunkUniversalForwarder\var\log\splunk\metrics.log.1":  Access is denied

As an admin, I can read/write into the same folder. Splunkd can write the log files Ok as the data and size is growing in each of the files. Any reason why access is denied when it tries to rename/unlink?

0 Karma

Path Finder

This one is confusing, it's happening on a number of machines here.

The sequence is (or should be):
metrics.log.5 gets deleted
metrics.log.4 is renamed to metrics.log.5
metrics.log.3 is renamed to metrics.log.4
metrics.log.2 is renamed to metrics.log.3
metrics.log.1 is renamed to metrics.log.2
metrics.log is renamed to metrics.log.1
a new metrics.log is created.

We are seeing all permissions removed on metrics.log.5
(i.e. an adminstrator has no permissions on the file to even inspect permissions)

This prevents the above sequence from occurring and our metrics.log files are getting larger and larger.

We do not understand what might be interfering with the permissions of the metrics.log.5 file, since all the other files are acciessible, manageable.

I am pretty sure we don't have people looking at metrics.log.5 with a notepad. It's also happening on a number of machines.

Can a splunk person comment on the sequence of actions taken by the UF when rolling out metrics.log.5 ?

We can't tell if something we have in place is occasionally interfering with the removal of it.

0 Karma

Super Champion

That usually happens when you have a lock on those files somehow.
I've seen it when using tail or notepad.
Make sure nothing is reading from your metrics.log as that's the one that can't be renamed.

Super Champion

nothing is reading the file other than Splunk UniversalForwader itself trying to send to Indexer

0 Karma

Super Champion

Can you try using Procexp to double check that?

If nothing is locking it according to Procexp, try restarting Splunk and it that works then it probably means Splunk was locking those files and that's not great. I would raise a support request but I guess you might be asked to replicate the problem and that might not be easy.

0 Karma

Super Champion

thank you for your assistance. I will hopefully raise a support request

0 Karma
Get Updates on the Splunk Community!

.conf24 | Day 0

Hello Splunk Community! My name is Chris, and I'm based in Canberra, Australia's capital, and I travelled for ...

Enhance Security Visibility with Splunk Enterprise Security 7.1 through Threat ...

 (view in My Videos)Struggling with alert fatigue, lack of context, and prioritization around security ...

Troubleshooting the OpenTelemetry Collector

  In this tech talk, you’ll learn how to troubleshoot the OpenTelemetry collector - from checking the ...