Installation

The minimum free disk space (5000MB) reached for /opt/splunk/var/run/splunk/dispatch.

rajiv_r
Explorer

I am using Splunk trail version and recently received the message "The minimum free disk space (5000MB) reached for /opt/splunk/var/run/splunk/dispatch.". I checked in splunk community and Splunk documents and as per the solution suggested i changed the free disc space to 20000 mb through splunk web. But still receiving the same message. And because of this error the indexing has been stopped and dashboard display no data.

Also made the changes in the main and the audit index by reducing the TSIDX file older than 30 days but still receiving the same message.

Can anyone please help me finding the solution as I am new in Splunk and not very proficient in Admin part. Do I need to increase the Instance size, currently I am using T3 Large(120 GB  ).

requesting every Splunk professional to suggest a solution

Labels (1)
0 Karma

rajiv_r
Explorer

HI  gucsello

i have deleted couple of folder from the dispatch directory. But still the error has not removed. It still showing dispatch directory is full error.

Shall i go ahead and delete each and every folder without any worry. Please suggest as all my dashbaord and reports are not working because of this issue.

Also if you can help me in using the clean dispatch command in a proper way. 

when i am running a command - splunkd clean-dispatch /opt/splunk/var/run/splunk/old-dispatch-jobs/-10d@d  it says splunkd: command not found. where i exactly do i need to run this command. Please guide me to few more step i am close to the solution

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @rajiv_r,

You didn't reach the maximum disk space for indexing but for dispatching, this meand that the problem is another.

To solve it, you have to:

  • stop Splunk;
  • go at /opt/splunk/var/run/splunk/dispatch;
  • delete the older folders;
  • start Splunk

Ciao.

Giuseppe

0 Karma

rajiv_r
Explorer

Thanks for your reply. 

I checked the dispatch directory it showing the below files

scheduler__nobody_c3BsdW5rX2luc3RydW1lbnRhdGlvbg__RMD5033d5cc4dca52033_at_1615777200_32
scheduler__nobody_c3BsdW5rX2luc3RydW1lbnRhdGlvbg__RMD514c5919b98bd054a_at_1615172400_29
scheduler__nobody_c3BsdW5rX2luc3RydW1lbnRhdGlvbg__RMD514c5919b98bd054a_at_1615777200_30

i used the clean dispatch command for moving the file to the new destination but its not working. Created a destination file in the same file system. But still its not working.

can you please throw some more light on this..as to how to resolve this

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @rajiv_r,

these forder contain the results of the user searches.

So you can delete the older ones without problems (after Splunk stop).

Ciao.

Giuseppe

rajiv_r
Explorer

i could not able to to delete the file from dispatch directory. The dispatch directory contains args.txt and, info.csv, search.logs etc . Whne i am running the command-  rm info.csv it says no such file or directory exists. 

I am also trying to run the clear dispatch command to move it into new folder but its not working. Created a new directory in the location where dispatch directory is stored. and running the below command

splunkd clean-dispatch /new/ -1month but its not working. I am new in working in this linux environment. Please help me to know where i am going wrong 

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @rajiv_r,

you haven't to delete single files, but full folders.

in other words yiou have to run rm -rf <folder_in_dispatch>

Choose the oldest.

Ciao.

Giuseppe

rajiv_r
Explorer

was doing a silly mistake while deleting that file. Sussesfully removed file from dispatch directory but the error is still coming. One more suggestion do in need to take a copy of these file to retrieve it in future. Or shall i go ahead and delete each one of them without any worry

Thanks in Advance

0 Karma