Splunk Enterprise
Highlighted

Getting search disk quota error and dispatch directory getting filled from unknown source.

New Member

Hello I have this dispatch directory getting filled by by RemoteStorageRetrieveIndexes_* directory getting created multiple times in a minute. I am not sure where this is coming from. I checked all the saved searches, alerts. I even recursively grepped the entire splunk config directory but found nothing defined by this name. I think this is causing issue with search disk quota being exhausted. What could be creating this directory? It only started happening recently.

Labels (1)
Tags (1)
0 Karma
Highlighted

Re: Getting search disk quota error and dispatch directory getting filled from unknown source.

Motivator

The dispatch directory is where Splunk stores search artifacts on the search heads, and is configured at the role level.
The default is 100MB, which is generally WAY too low, but is intended (I think) to be a safety measure, of sorts.

Searches that return raw events are sent from the indexers to the search head(s) and stored there, then the SH's do the parsing and displaying of results. If the default dispatch directory is too small to store the results returned from the indexers then you'll encounter this error.

In my experience, it has been good practice to increase the directory size limit for admin to a far, far greater size. Generally I set it to 30GB, but your environment will differ. For individual users/roles, it is a bit of a formula. You need to understand what your users are searching, and the size of the artifacts that are generated.

The directory size can be found on the search head, or search head cluster node, at Settings > Account Settings > Roles (choose the role).

Worth noting, this size, and the concurrent search limits, on the DMC in particular need to be increased substantially, in my experience.

View solution in original post

0 Karma
Highlighted

Re: Getting search disk quota error and dispatch directory getting filled from unknown source.

New Member

Thank you. I was able to fix it by making up some space in the filesystem and deleting some huge saved searches. It looks like Splunk creates these RemoteStorageRetrieveIndexes_* directories when the disk space falls below desired value.

0 Karma
Highlighted

Re: Getting search disk quota error and dispatch directory getting filled from unknown source.

Motivator

Awesome, glad this helped you!

0 Karma
Speak Up for Splunk Careers!

We want to better understand the impact Splunk experience and expertise has has on individuals' careers, and help highlight the growing demand for Splunk skills.