Splunk Enterprise

Getting search disk quota error and dispatch directory getting filled from unknown source.

mahars01
New Member

Hello I have this dispatch directory getting filled by by RemoteStorageRetrieveIndexes_* directory getting created multiple times in a minute. I am not sure where this is coming from. I checked all the saved searches, alerts. I even recursively grepped the entire splunk config directory but found nothing defined by this name. I think this is causing issue with search disk quota being exhausted. What could be creating this directory? It only started happening recently.

Labels (1)
Tags (1)
0 Karma
1 Solution

codebuilder
Influencer

The dispatch directory is where Splunk stores search artifacts on the search heads, and is configured at the role level.
The default is 100MB, which is generally WAY too low, but is intended (I think) to be a safety measure, of sorts.

Searches that return raw events are sent from the indexers to the search head(s) and stored there, then the SH's do the parsing and displaying of results. If the default dispatch directory is too small to store the results returned from the indexers then you'll encounter this error.

In my experience, it has been good practice to increase the directory size limit for admin to a far, far greater size. Generally I set it to 30GB, but your environment will differ. For individual users/roles, it is a bit of a formula. You need to understand what your users are searching, and the size of the artifacts that are generated.

The directory size can be found on the search head, or search head cluster node, at Settings > Account Settings > Roles (choose the role).

Worth noting, this size, and the concurrent search limits, on the DMC in particular need to be increased substantially, in my experience.

----
An upvote would be appreciated and Accept Solution if it helps!

View solution in original post

0 Karma

mahars01
New Member

Thank you. I was able to fix it by making up some space in the filesystem and deleting some huge saved searches. It looks like Splunk creates these RemoteStorageRetrieveIndexes_* directories when the disk space falls below desired value.

0 Karma

codebuilder
Influencer

Awesome, glad this helped you!

----
An upvote would be appreciated and Accept Solution if it helps!
0 Karma

codebuilder
Influencer

The dispatch directory is where Splunk stores search artifacts on the search heads, and is configured at the role level.
The default is 100MB, which is generally WAY too low, but is intended (I think) to be a safety measure, of sorts.

Searches that return raw events are sent from the indexers to the search head(s) and stored there, then the SH's do the parsing and displaying of results. If the default dispatch directory is too small to store the results returned from the indexers then you'll encounter this error.

In my experience, it has been good practice to increase the directory size limit for admin to a far, far greater size. Generally I set it to 30GB, but your environment will differ. For individual users/roles, it is a bit of a formula. You need to understand what your users are searching, and the size of the artifacts that are generated.

The directory size can be found on the search head, or search head cluster node, at Settings > Account Settings > Roles (choose the role).

Worth noting, this size, and the concurrent search limits, on the DMC in particular need to be increased substantially, in my experience.

----
An upvote would be appreciated and Accept Solution if it helps!

View solution in original post

0 Karma
Register for .conf21 Now! Go Vegas or Go Virtual!

How will you .conf21? You decide! Go in-person in Las Vegas, 10/18-10/21, or go online with .conf21 Virtual, 10/19-10/20.