Monitoring Splunk

Troubleshooting disk usage peak on the Search Head server

kvnpichon
Path Finder

Hello,

This is my architecture :

  • dedicated indexers (multiple servers on main site)
  • dedicated search head (1 serveron main site)
  • dedicated management server (1 server on main site)
  • dedicated syslog/forwarders (1 server per site)

I have an issue with my Search Head. When I check the DMC I can see there are disk usage peaks sometimes and it immediatly goes down.

For example, the last peak is today, started at 10:15 and it goes down to 13:45.

Meanwhile I don't understand this peaks and where did the data came from ?

I checked logs in Splunk but I have no clues.

I don't know if I miust check it in Splunk or in Linux.

Hope you can help me Splunkers,

Regards

0 Karma

richgalloway
SplunkTrust
SplunkTrust

Disk usage in search heads often can be attributed to search results returned from indexers.  Results are retained in the dispatch directory for a short period, which is why usage rises and falls.

---
If this reply helps you, Karma would be appreciated.
0 Karma

somesoni2
Revered Legend

You might be getting peaks in disk usage because of the search activities going on during that time. Every search will have a dispatch directory to store search artifacts and if too many searches are running, the size of dispatch directory will go high. Splunk does cleanup the dispatch directory when the jobs expire , so that will explain how peaks go away after certain time.

If you can log onto search head server, and look the size of dispatch directory ($SPLUNK_HOME/var/run/splunk/dispatch) before and during those peak hours.

0 Karma

kvnpichon
Path Finder

OK, I will try to check it and I will be back to the post.

Thanks for the reply @richgalloway  and @somesoni2 

If the issue comes from the dispatch folder, how can I reduce the max size of the folder in order to solve the issue ? or maybe there is a better way to do this ?

Is there a history to check the size of the folder or something directly in Splunk ?

0 Karma

richgalloway
SplunkTrust
SplunkTrust

You don't want to reduce the maximum size of the dispatch folder.  Doing so means you might run out of space for search results and be unable to display them.

A better approach is to reduce any or all of the following:

  • The number of searches run.
  • The retention time of search results, using dispatch.ttl in savedsearches.conf
  • The amount of data returned for each search, using fields and other search commands.
---
If this reply helps you, Karma would be appreciated.
0 Karma
Get Updates on the Splunk Community!

.conf24 | Day 0

Hello Splunk Community! My name is Chris, and I'm based in Canberra, Australia's capital, and I travelled for ...

Enhance Security Visibility with Splunk Enterprise Security 7.1 through Threat ...

(view in My Videos)Struggling with alert fatigue, lack of context, and prioritization around security ...

Troubleshooting the OpenTelemetry Collector

  In this tech talk, you’ll learn how to troubleshoot the OpenTelemetry collector - from checking the ...