Monitoring Splunk

Error Spamming Splunkd.log Error Process_Search

Motivator

I'm getting the following spammed hundreds of thousands of time in my log splunkd.log file

ERROR ProcessDispatchedSearch - PROCESS_SEARCH - Error opening C:\Program Files\Splunk\var\run\splunk\dispatch\{insertDirectoryNameHere}\search.log: The operation completed successfully.

This is causing my indexer to become congested and frozen.
Attempts to restart splunk fail and when running 'splunk restart' on the server says that port 8090 is in use and it will not allow me to start splunk back up.

Restarting the machine brings the indexes back to life however this does not last long when using splunk and looking at dashboards etc.

Splunk is running on a VM with Windows server 2008 R2

0 Karma
1 Solution

Path Finder

If you're running anti-virus, that might be causing it. To get "The operation completed successfully" when opening a file suggests some strange condition that "shouldn't happen", exactly the sort of thing that AV scanners like to cause. They also love to lock files at inopportune times, which can cause open's to fail.

View solution in original post

Motivator

Somehow the directories (and all the child directories/files) of "var/run" and "var/spool" lost all permissions. Giving Permissions to System/Administrator of the folders and their child folders may have fixed my issue.

Edit:
It appears that all of these files are being created automatically with no permissions, When splunk tries to read them it can't find them. This appears to be a bug in Splunk and is happening on my own machine and my Dev Machine.

0 Karma

Path Finder

If you're running anti-virus, that might be causing it. To get "The operation completed successfully" when opening a file suggests some strange condition that "shouldn't happen", exactly the sort of thing that AV scanners like to cause. They also love to lock files at inopportune times, which can cause open's to fail.

View solution in original post

Motivator

I believe you are correct, our antivirus is locking many splunk files causing splunk to not work correctly. Since I'm pretty sure this is the issue I am going to mark this as answer, Thanks!

0 Karma
State of Splunk Careers

Access the Splunk Careers Report to see real data that shows how Splunk mastery increases your value and job satisfaction.

Find out what your skills are worth!