topic Re: Error Spamming Splunkd.log Error Process_Search in Monitoring Splunk

Error Spamming Splunkd.log Error Process_Search

aelliott — Tue, 03 Dec 2013 17:47:58 GMT

I'm getting the following spammed hundreds of thousands of time in my log splunkd.log file

ERROR ProcessDispatchedSearch - PROCESS_SEARCH - Error opening C:\Program Files\Splunk\var\run\splunk\dispatch\{insertDirectoryNameHere}\search.log: The operation completed successfully.

This is causing my indexer to become congested and frozen.
Attempts to restart splunk fail and when running 'splunk restart' on the server says that port 8090 is in use and it will not allow me to start splunk back up.

Restarting the machine brings the indexes back to life however this does not last long when using splunk and looking at dashboards etc.

Splunk is running on a VM with Windows server 2008 R2

Re: Error Spamming Splunkd.log Error Process_Search

sciurus — Thu, 05 Dec 2013 08:56:01 GMT

If you're running anti-virus, that might be causing it. To get "The operation completed successfully" when opening a file suggests some strange condition that "shouldn't happen", exactly the sort of thing that AV scanners like to cause. They also love to lock files at inopportune times, which can cause open's to fail.

Re: Error Spamming Splunkd.log Error Process_Search

aelliott — Thu, 05 Dec 2013 15:31:01 GMT

Somehow the directories (and all the child directories/files) of "var/run" and "var/spool" lost all permissions. Giving Permissions to System/Administrator of the folders and their child folders may have fixed my issue.

Edit:
It appears that all of these files are being created automatically with no permissions, When splunk tries to read them it can't find them. This appears to be a bug in Splunk and is happening on my own machine and my Dev Machine.

Re: Error Spamming Splunkd.log Error Process_Search

aelliott — Fri, 06 Dec 2013 15:22:15 GMT

I believe you are correct, our antivirus is locking many splunk files causing splunk to not work correctly. Since I'm pretty sure this is the issue I am going to mark this as answer, Thanks!