Splunk Forwarder reading log on UINC share needs extra NTFS permissions

[davyyd]

We are running Splunk Forwarder version 6.2.1 on a Windows 2012R2 server as "Local System". We have IIS also running as "Local System". IIS is logging to a UNC path successfully. The permissions on the UNC share and NTFS are allowing the domain computer account full access (i.e. domain\Server01$ has permissions on the share on filesystem). This is sufficient to allow IIS to log, but the Splunk forwarder cannot read the contents of the log file. I know it can read the contents of the directory, because the stanza in inputs.conf contains regex, and the error entry in splunkd.log states the actual file name. When I add either Administrators, SYSTEM, or Everyone to the NTFS permissions of the log file, Splunk is able to read it.

Here is the stanza in inputs.conf:
[monitor://\fqdn.of.server\LogShare[0-9a-zA-Z-]+*[0-9a-zA-Z-]+\W3SVC[0-9]+\u_ex[0-9][0-9][0-9][0-9][0-9][0-9].log*]
sourcetype = iis
ignoreOlderThan = 1d
disabled = 0
index = main

And here is a sample entry from splunkd.log when the service is started, but NTFS is granting the machine account full access to the Log:
05-04-2015 14:47:58.090 -0700 WARN TailingProcessor - Insufficient permissions to read file='\fqdn.of.server\LogShare\SiteName\ServerName\W3SVC9\u_ex150504.log' (hint: Access is denied.)

And here is an entry from splunkd.log when the same log file has "SYSTEM" (or Administrators, or Everyone) add to the NTFS permissions:
05-11-2015 16:27:11.681 -0700 WARN FileClassifierManager - The CHECK_FOR_HEADER setting is deprecated - INDEXED_EXTRACTIONS should be used instead for file="\fqdn.of.server\LogShare\SiteName\ServerName\W3SVC5\u_ex150511.log".
05-11-2015 16:27:11.681 -0700 WARN FileClassifierManager - Detected INDEXED_EXTRACTIONS setting - disabling deprecated CHECK_FOR_HEADER setting for file="\fqdn.of.server\LogShare\SiteName\ServerName\W3SVC5\u_ex150511.log".
05-11-2015 16:27:11.681 -0700 INFO WatchedFile - Resetting fd to re-extract header.

Only at this point will the forwarder begin sending IIS Log entries to the indexer.

From the point of view of Share and NTFS permissions, this does not make sense to me. Any process running as Local System on the IIS server should be able to access the log file as the computer account. This is proven by both IIS being able to write the log and by the Splunk Forwarder being able to see that the log exists. Can anyone help me understand why it is necessary that the additional permissions be added to NTFS to enable the forwarder to work?

[dwaddle]

(not the answer you are seeking, but...) I would suggest that you not have forwarders do remote UNC collection of data. It can be vastly inefficient. Put a forwarder directly on the host you are writing logs to, and enjoy much better performance and fewer headaches.

[davyyd]

Thanks for the contribution dwaddle, and I agree. However, in our case the Web files are stored on a (non-windows) NAS. In order to eliminate the NAS as a possible problem, I replicated the issue on a Windows server / share, but the final product will need to work on our NAS.