Monitoring Splunk

Health Status of Splunkd: Why does the Disk Space show RED?

rwrettig
New Member

On the top of my screen is a red ! - after clicking on it it tells me my disk space is in the red
! Disk Space
-Root Cause
--The diskspace remaining =5434 has breached the red threshold for filesystems=(opt/splunk/var/splunk/audit/db) )
--Last 50 related messages

So I'm out of disk space I presume, so how do i fix it? I have a free setup (I'm a student) with Splunk and an instance on AWS.

Please help!

0 Karma

woodcock
Esteemed Legend

Limits for controlling disk space in Splunk can be changed

The relevant stanza and parameter of interest in server.conf is:

[diskUsage]
minFreeSpace = <num>

For more details, check here:
http://docs.splunk.com/Documentation/Splunk/latest/Indexer/Setlimitsondiskusage

This can be changed on any Splunk installations as explained on the online documentation:

for all installations, including forwarders, you must have a minimum of 5GB of hard disk space available in addition to the space required for any indexes.

The default is 5000(mb) and this value can be changed as explained before.

For more details, check here:
http://docs.splunk.com/Documentation/Splunk/latest/Installation/Systemrequirements#Recommended_hardw...

woodcock
Esteemed Legend

Go to Settings -> Monitoring Console -> Health Checks and run all the checks. Drill into the red ones and research the details.

0 Karma

rwrettig
New Member

Thank you for replying!

Ran a Health Check.

4 parts are in yellow

  1. Assessment of server ulimits - One or more Splunk instances are running on a host that has one or more resource limits set below official recommendations.
  2. Linux kernel transparent huge pages - One or more Splunk instances are running on a host that has kernel transparent huge pages enabled. This can significantly reduce performance and is against best practice.
  3. Search scheduler skip ratio - Scheduled searches are being skipped on one or more search heads.
  4. System hardware provisioning assessment - One or more hosts has returned CPU or memory specifications that fall below reference hardware recommendations. This might adversely affect indexing or search performance.

I only have one instance running on AWS.

0 Karma

woodcock
Esteemed Legend

Is this a lab? Are you trying to learn or get something hardened for production?

0 Karma

rwrettig
New Member

Learning Splunk!

0 Karma
.conf21 CFS Extended through 5/20!

Don't miss your chance
to share your Splunk
wisdom in-person or
virtually at .conf21!

Call for Speakers has
been extended through
Thursday, 5/20!