Error Spamming Splunkd.log Error Process_Search

[aelliott]

I'm getting the following spammed hundreds of thousands of time in my log splunkd.log file

ERROR ProcessDispatchedSearch - PROCESS_SEARCH - Error opening C:\Program Files\Splunk\var\run\splunk\dispatch\{insertDirectoryNameHere}\search.log: The operation completed successfully.

This is causing my indexer to become congested and frozen.
Attempts to restart splunk fail and when running 'splunk restart' on the server says that port 8090 is in use and it will not allow me to start splunk back up.

Restarting the machine brings the indexes back to life however this does not last long when using splunk and looking at dashboards etc.

Splunk is running on a VM with Windows server 2008 R2

[aelliott]

Somehow the directories (and all the child directories/files) of "var/run" and "var/spool" lost all permissions. Giving Permissions to System/Administrator of the folders and their child folders may have fixed my issue.

Edit:
It appears that all of these files are being created automatically with no permissions, When splunk tries to read them it can't find them. This appears to be a bug in Splunk and is happening on my own machine and my Dev Machine.

[sciurus · Accepted solution]

If you're running anti-virus, that might be causing it. To get "The operation completed successfully" when opening a file suggests some strange condition that "shouldn't happen", exactly the sort of thing that AV scanners like to cause. They also love to lock files at inopportune times, which can cause open's to fail.

[aelliott]

I believe you are correct, our antivirus is locking many splunk files causing splunk to not work correctly. Since I'm pretty sure this is the issue I am going to mark this as answer, Thanks!