Splunk Enterprise

Getting search disk quota error and dispatch directory getting filled from unknown source.

mahars01
Explorer

Hello I have this dispatch directory getting filled by by RemoteStorageRetrieveIndexes_* directory getting created multiple times in a minute. I am not sure where this is coming from. I checked all the saved searches, alerts. I even recursively grepped the entire splunk config directory but found nothing defined by this name. I think this is causing issue with search disk quota being exhausted. What could be creating this directory? It only started happening recently.

Labels (1)
Tags (1)
1 Solution

codebuilder
SplunkTrust
SplunkTrust

The dispatch directory is where Splunk stores search artifacts on the search heads, and is configured at the role level.
The default is 100MB, which is generally WAY too low, but is intended (I think) to be a safety measure, of sorts.

Searches that return raw events are sent from the indexers to the search head(s) and stored there, then the SH's do the parsing and displaying of results. If the default dispatch directory is too small to store the results returned from the indexers then you'll encounter this error.

In my experience, it has been good practice to increase the directory size limit for admin to a far, far greater size. Generally I set it to 30GB, but your environment will differ. For individual users/roles, it is a bit of a formula. You need to understand what your users are searching, and the size of the artifacts that are generated.

The directory size can be found on the search head, or search head cluster node, at Settings > Account Settings > Roles (choose the role).

Worth noting, this size, and the concurrent search limits, on the DMC in particular need to be increased substantially, in my experience.

----
An upvote would be appreciated and Accept Solution if it helps!

View solution in original post

mahars01
Explorer

Thank you. I was able to fix it by making up some space in the filesystem and deleting some huge saved searches. It looks like Splunk creates these RemoteStorageRetrieveIndexes_* directories when the disk space falls below desired value.

0 Karma

codebuilder
SplunkTrust
SplunkTrust

Awesome, glad this helped you!

----
An upvote would be appreciated and Accept Solution if it helps!
0 Karma

codebuilder
SplunkTrust
SplunkTrust

The dispatch directory is where Splunk stores search artifacts on the search heads, and is configured at the role level.
The default is 100MB, which is generally WAY too low, but is intended (I think) to be a safety measure, of sorts.

Searches that return raw events are sent from the indexers to the search head(s) and stored there, then the SH's do the parsing and displaying of results. If the default dispatch directory is too small to store the results returned from the indexers then you'll encounter this error.

In my experience, it has been good practice to increase the directory size limit for admin to a far, far greater size. Generally I set it to 30GB, but your environment will differ. For individual users/roles, it is a bit of a formula. You need to understand what your users are searching, and the size of the artifacts that are generated.

The directory size can be found on the search head, or search head cluster node, at Settings > Account Settings > Roles (choose the role).

Worth noting, this size, and the concurrent search limits, on the DMC in particular need to be increased substantially, in my experience.

----
An upvote would be appreciated and Accept Solution if it helps!
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...