Troubleshooting disk usage peak on the Search Head server

[kvnpichon]

Hello,

This is my architecture :

• dedicated indexers (multiple servers on main site)
• dedicated search head (1 serveron main site)
• dedicated management server (1 server on main site)
• dedicated syslog/forwarders (1 server per site)

I have an issue with my Search Head. When I check the DMC I can see there are disk usage peaks sometimes and it immediatly goes down.

For example, the last peak is today, started at 10:15 and it goes down to 13:45.

Meanwhile I don't understand this peaks and where did the data came from ?

I checked logs in Splunk but I have no clues.

I don't know if I miust check it in Splunk or in Linux.

Hope you can help me Splunkers,

Regards

[richgalloway]

Disk usage in search heads often can be attributed to search results returned from indexers.  Results are retained in the dispatch directory for a short period, which is why usage rises and falls.

[somesoni2]

You might be getting peaks in disk usage because of the search activities going on during that time. Every search will have a dispatch directory to store search artifacts and if too many searches are running, the size of dispatch directory will go high. Splunk does cleanup the dispatch directory when the jobs expire , so that will explain how peaks go away after certain time.

If you can log onto search head server, and look the size of dispatch directory ($SPLUNK_HOME/var/run/splunk/dispatch) before and during those peak hours.

[kvnpichon]

OK, I will try to check it and I will be back to the post.

Thanks for the reply @richgalloway  and @somesoni2 

If the issue comes from the dispatch folder, how can I reduce the max size of the folder in order to solve the issue ? or maybe there is a better way to do this ?

Is there a history to check the size of the folder or something directly in Splunk ?

[richgalloway]

You don't want to reduce the maximum size of the dispatch folder.  Doing so means you might run out of space for search results and be unable to display them.

A better approach is to reduce any or all of the following:

• The number of searches run.
• The retention time of search results, using dispatch.ttl in savedsearches.conf
• The amount of data returned for each search, using fields and other search commands.