Monitoring Splunk

Error Spamming Splunkd.log Error Process_Search

aelliott
Motivator

I'm getting the following spammed hundreds of thousands of time in my log splunkd.log file

ERROR ProcessDispatchedSearch - PROCESS_SEARCH - Error opening C:\Program Files\Splunk\var\run\splunk\dispatch\{insertDirectoryNameHere}\search.log: The operation completed successfully.

This is causing my indexer to become congested and frozen.
Attempts to restart splunk fail and when running 'splunk restart' on the server says that port 8090 is in use and it will not allow me to start splunk back up.

Restarting the machine brings the indexes back to life however this does not last long when using splunk and looking at dashboards etc.

Splunk is running on a VM with Windows server 2008 R2

0 Karma
1 Solution

sciurus
Path Finder

If you're running anti-virus, that might be causing it. To get "The operation completed successfully" when opening a file suggests some strange condition that "shouldn't happen", exactly the sort of thing that AV scanners like to cause. They also love to lock files at inopportune times, which can cause open's to fail.

View solution in original post

aelliott
Motivator

Somehow the directories (and all the child directories/files) of "var/run" and "var/spool" lost all permissions. Giving Permissions to System/Administrator of the folders and their child folders may have fixed my issue.

Edit:
It appears that all of these files are being created automatically with no permissions, When splunk tries to read them it can't find them. This appears to be a bug in Splunk and is happening on my own machine and my Dev Machine.

0 Karma

sciurus
Path Finder

If you're running anti-virus, that might be causing it. To get "The operation completed successfully" when opening a file suggests some strange condition that "shouldn't happen", exactly the sort of thing that AV scanners like to cause. They also love to lock files at inopportune times, which can cause open's to fail.

aelliott
Motivator

I believe you are correct, our antivirus is locking many splunk files causing splunk to not work correctly. Since I'm pretty sure this is the issue I am going to mark this as answer, Thanks!

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...