topic Re: DOWN! Error: skipped indexing of internal audit event will keep dropping events until indexer congestion is remedied. Check disk space and other issues that may cause indexer to block in Splunk Search

DOWN! Error: skipped indexing of internal audit event will keep dropping events until indexer congestion is remedied. Check disk space and other issues that may cause indexer to block

kholleran — Fri, 15 Oct 2010 20:30:30 GMT

Good morning,

I am suddenly receiving this error and not able to index:

skipped indexing of internal audit event will keep dropping events until indexer congestion is remedied. Check disk space and other issues that may cause indexer to block

The other day I received this error:

Applying indexing throttle for defaultdb\db because bucket has too many tsidx files, is your splunk-optimize working?

I have recently upgraded from 4.1.3 to 4.1.5. There was no immediate change but I did start using FSChange to monitor some directories.

I removed FSChange stanzas that I added from the inputs.conf and restarted and I am still having the issue, though the warning moved back to the second error.

In splunkd.log I see:

10-15-2010 08:20:18.131 ERROR DispatchCommand - Failed to start the search process.  
10-15-2010 08:20:18.162 WARN  DispatchCommand - The system is approaching the maximum number of historical searches that can be run concurrently. current=7 maximum=8  
10-15-2010 08:20:18.193 ERROR DispatchCommand - Failed to start the search process.  
10-15-2010 08:20:19.850 ERROR DispatchCommand - The maximum number of historical concurrent system-wide searches has been reached. current=8 maximum=8 Search not executed! SearchId=scheduler__nobody__windows_d2luX2V2ZW50bG9nX2NvdW50X3N1bV9pbmRleA_at_1287148800_967218771  
10-15-2010 08:20:19.896 ERROR SearchScheduler - The maximum number of historical concurrent system-wide searches has been reached. current=8 maximum=8 Search not executed!   SearchId=scheduler__nobody__windows_d2luX2V2ZW50bG9nX2NvdW50X3N1bV9pbmRleA_at_1287148800_967218771  
10-15-2010 08:20:23.193 WARN  timeinvertedIndex - splunk-optimize failed to start for index D:\Splunk_Data\var\defaultdb\db\hot_v1_16 : The session was canceled.  
10-15-2010 08:20:23.193 WARN  timeinvertedIndex - splunk-optimize failed to start for index D:\Splunk_Data\var\defaultdb\db\hot_v1_19 : The session was canceled.

I am not sure if it is related. Perhaps with all my alerts that run at various intervals (10 min, 15 min, 20 min, 30 min) I am eclipsing 8. Would that cause the errors regarding not indexing?

I am currently not able to view any data for the last two days.

Thanks for any help!

Kevin

Re: DOWN! Error: skipped indexing of internal audit event will keep dropping events until indexer congestion is remedied. Check disk space and other issues that may cause indexer to block

Lowell — Fri, 15 Oct 2010 20:42:28 GMT

For a "down" kind of scenario like this, it may be best to contact splunk support. Email them with a link to this page, run the "splunk diag" utility, upload the diag file to your case, then call the splunk support phone # to get in contact with someone quickly.

Things I would check:

Disk space on all of your partitions. If your space is too low, then that will cause indexing and searching problems.
Verify that "splunk-optimize" has been running. If you see a large number of *.tsidx files in your buckets, you can simply run splunk-optimize /path/to/your/bucket to force this process to run.
Try disabling some non-essential scheduled saved searches and see if that helps relieve the problem. (You probably don't want to do this for summary-indexing saved searches, if it can be avoided.)

Re: DOWN! Error: skipped indexing of internal audit event will keep dropping events until indexer congestion is remedied. Check disk space and other issues that may cause indexer to block

kholleran — Fri, 15 Oct 2010 22:50:23 GMT

I have an open case and am working with them. Disk space is fine, I disabled a scheduled saved search, but just the one I added prior to this problem occurring.

I am looking into #2. Thanks very much for all your help.

Re: DOWN! Error: skipped indexing of internal audit event will keep dropping events until indexer congestion is remedied. Check disk space and other issues that may cause indexer to block

kholleran — Tue, 19 Oct 2010 20:36:59 GMT

Unfortunately, those did not work and I have not heard back in a couple days regarding my case with Splunk. Any other thoughts as I am completely down. If this goes on much longer I may have to downgrade back to 4.1.3, even if that means ripping out and re-setting up the installation. Thanks.

Re: DOWN! Error: skipped indexing of internal audit event will keep dropping events until indexer congestion is remedied. Check disk space and other issues that may cause indexer to block

Lowell — Wed, 20 Oct 2010 21:44:14 GMT

I suggest calling splunk support. Copied from the "Contact Us" web page: If you have purchased Enterprise Support, please call the Enterprise Support line at +1 415.848.8400 option 3.

Re: DOWN! Error: skipped indexing of internal audit event will keep dropping events until indexer congestion is remedied. Check disk space and other issues that may cause indexer to block

chicodeme — Thu, 20 Oct 2011 16:12:12 GMT

What did it end up being?

Re: DOWN! Error: skipped indexing of internal audit event will keep dropping events until indexer congestion is remedied. Check disk space and other issues that may cause indexer to block

chris_lewis — Wed, 27 Mar 2013 17:22:43 GMT

What was the resolution to this?!?

Re: DOWN! Error: skipped indexing of internal audit event will keep dropping events until indexer congestion is remedied. Check disk space and other issues that may cause indexer to block

MegSplunk — Thu, 23 Jan 2014 13:20:38 GMT

Were you able to resolve this?

Re: DOWN! Error: skipped indexing of internal audit event will keep dropping events until indexer congestion is remedied. Check disk space and other issues that may cause indexer to block

bosburn_splunk — Thu, 23 Jan 2014 14:20:43 GMT

Best bet is to contact support and let them work through it with you as there can be many causes of it.

Re: DOWN! Error: skipped indexing of internal audit event will keep dropping events until indexer congestion is remedied. Check disk space and other issues that may cause indexer to block

hanijamal — Thu, 21 Apr 2016 15:59:24 GMT

also having the same issue.. not a disk space problem either on my end.. any resolution to this?

Re: DOWN! Error: skipped indexing of internal audit event will keep dropping events until indexer congestion is remedied. Check disk space and other issues that may cause indexer to block

woodcock — Sat, 22 Oct 2016 14:01:25 GMT

I had this problem recently and it was for a tricky/silly reason. I got tired of the dispatch directory being tied to the root volume and getting The minimum free disk space (5000MB) reached for /opt/splunk/var/run/splunk/dispatch errors, so i created a 10G volume and mounted it over dispatch BUT I neglected to make it writable by the user running splunkd (i.e. "splunk"). In such a situation, 14 searches will start, but not really, and none will be able to complete so you get hung. I discovered the problem by going to the search head CLI and doing this (because I could not search against _*):

tail -f $SPLUNK_HOME/var/log/splunk/*

Very quickly I saw logs like this:

10-21-2016 12:02:10.208 -0400 ERROR SearchScheduler - failed to rm -r /opt/splunk/var/run/splunk/dispatch/scheduler__nobody_c3BsdW5rX21vbml0b3JpbmdfY29uc29sZQ__RMD54740dfff07b17ef1_at_1477065699_0: No such file or directory

In other words, it was trying to remove files that it was not able to create. OOPS! A simple chmod later and all was good again.