Getting search disk quota error and dispatch directory getting filled from unknown source.

[mahars01]

Hello I have this dispatch directory getting filled by by RemoteStorageRetrieveIndexes_* directory getting created multiple times in a minute. I am not sure where this is coming from. I checked all the saved searches, alerts. I even recursively grepped the entire splunk config directory but found nothing defined by this name. I think this is causing issue with search disk quota being exhausted. What could be creating this directory? It only started happening recently.

[mahars01]

Thank you. I was able to fix it by making up some space in the filesystem and deleting some huge saved searches. It looks like Splunk creates these RemoteStorageRetrieveIndexes_* directories when the disk space falls below desired value.

[codebuilder]

Awesome, glad this helped you!

[codebuilder · Accepted solution]

The dispatch directory is where Splunk stores search artifacts on the search heads, and is configured at the role level.
The default is 100MB, which is generally WAY too low, but is intended (I think) to be a safety measure, of sorts.

Searches that return raw events are sent from the indexers to the search head(s) and stored there, then the SH's do the parsing and displaying of results. If the default dispatch directory is too small to store the results returned from the indexers then you'll encounter this error.

In my experience, it has been good practice to increase the directory size limit for admin to a far, far greater size. Generally I set it to 30GB, but your environment will differ. For individual users/roles, it is a bit of a formula. You need to understand what your users are searching, and the size of the artifacts that are generated.

The directory size can be found on the search head, or search head cluster node, at Settings > Account Settings > Roles (choose the role).

Worth noting, this size, and the concurrent search limits, on the DMC in particular need to be increased substantially, in my experience.