Splunk Enterprise

Getting search disk quota error and dispatch directory getting filled from unknown source.

mahars01
Explorer

Hello I have this dispatch directory getting filled by by RemoteStorageRetrieveIndexes_* directory getting created multiple times in a minute. I am not sure where this is coming from. I checked all the saved searches, alerts. I even recursively grepped the entire splunk config directory but found nothing defined by this name. I think this is causing issue with search disk quota being exhausted. What could be creating this directory? It only started happening recently.

Labels (1)
Tags (1)
1 Solution

codebuilder
SplunkTrust
SplunkTrust

The dispatch directory is where Splunk stores search artifacts on the search heads, and is configured at the role level.
The default is 100MB, which is generally WAY too low, but is intended (I think) to be a safety measure, of sorts.

Searches that return raw events are sent from the indexers to the search head(s) and stored there, then the SH's do the parsing and displaying of results. If the default dispatch directory is too small to store the results returned from the indexers then you'll encounter this error.

In my experience, it has been good practice to increase the directory size limit for admin to a far, far greater size. Generally I set it to 30GB, but your environment will differ. For individual users/roles, it is a bit of a formula. You need to understand what your users are searching, and the size of the artifacts that are generated.

The directory size can be found on the search head, or search head cluster node, at Settings > Account Settings > Roles (choose the role).

Worth noting, this size, and the concurrent search limits, on the DMC in particular need to be increased substantially, in my experience.

----
An upvote would be appreciated and Accept Solution if it helps!

View solution in original post

mahars01
Explorer

Thank you. I was able to fix it by making up some space in the filesystem and deleting some huge saved searches. It looks like Splunk creates these RemoteStorageRetrieveIndexes_* directories when the disk space falls below desired value.

0 Karma

codebuilder
SplunkTrust
SplunkTrust

Awesome, glad this helped you!

----
An upvote would be appreciated and Accept Solution if it helps!
0 Karma

codebuilder
SplunkTrust
SplunkTrust

The dispatch directory is where Splunk stores search artifacts on the search heads, and is configured at the role level.
The default is 100MB, which is generally WAY too low, but is intended (I think) to be a safety measure, of sorts.

Searches that return raw events are sent from the indexers to the search head(s) and stored there, then the SH's do the parsing and displaying of results. If the default dispatch directory is too small to store the results returned from the indexers then you'll encounter this error.

In my experience, it has been good practice to increase the directory size limit for admin to a far, far greater size. Generally I set it to 30GB, but your environment will differ. For individual users/roles, it is a bit of a formula. You need to understand what your users are searching, and the size of the artifacts that are generated.

The directory size can be found on the search head, or search head cluster node, at Settings > Account Settings > Roles (choose the role).

Worth noting, this size, and the concurrent search limits, on the DMC in particular need to be increased substantially, in my experience.

----
An upvote would be appreciated and Accept Solution if it helps!
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...