Monitoring Splunk

Error Spamming Splunkd.log Error Process_Search

aelliott
Motivator

I'm getting the following spammed hundreds of thousands of time in my log splunkd.log file

ERROR ProcessDispatchedSearch - PROCESS_SEARCH - Error opening C:\Program Files\Splunk\var\run\splunk\dispatch\{insertDirectoryNameHere}\search.log: The operation completed successfully.

This is causing my indexer to become congested and frozen.
Attempts to restart splunk fail and when running 'splunk restart' on the server says that port 8090 is in use and it will not allow me to start splunk back up.

Restarting the machine brings the indexes back to life however this does not last long when using splunk and looking at dashboards etc.

Splunk is running on a VM with Windows server 2008 R2

0 Karma
1 Solution

sciurus
Path Finder

If you're running anti-virus, that might be causing it. To get "The operation completed successfully" when opening a file suggests some strange condition that "shouldn't happen", exactly the sort of thing that AV scanners like to cause. They also love to lock files at inopportune times, which can cause open's to fail.

View solution in original post

aelliott
Motivator

Somehow the directories (and all the child directories/files) of "var/run" and "var/spool" lost all permissions. Giving Permissions to System/Administrator of the folders and their child folders may have fixed my issue.

Edit:
It appears that all of these files are being created automatically with no permissions, When splunk tries to read them it can't find them. This appears to be a bug in Splunk and is happening on my own machine and my Dev Machine.

0 Karma

sciurus
Path Finder

If you're running anti-virus, that might be causing it. To get "The operation completed successfully" when opening a file suggests some strange condition that "shouldn't happen", exactly the sort of thing that AV scanners like to cause. They also love to lock files at inopportune times, which can cause open's to fail.

aelliott
Motivator

I believe you are correct, our antivirus is locking many splunk files causing splunk to not work correctly. Since I'm pretty sure this is the issue I am going to mark this as answer, Thanks!

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...