Monitoring Splunk

Health Status of Splunkd: Why does the Disk Space show RED?

rwrettig
New Member

On the top of my screen is a red ! - after clicking on it it tells me my disk space is in the red
! Disk Space
-Root Cause
--The diskspace remaining =5434 has breached the red threshold for filesystems=(opt/splunk/var/splunk/audit/db) )
--Last 50 related messages

So I'm out of disk space I presume, so how do i fix it? I have a free setup (I'm a student) with Splunk and an instance on AWS.

Please help!

0 Karma

woodcock
Esteemed Legend

Limits for controlling disk space in Splunk can be changed

The relevant stanza and parameter of interest in server.conf is:

[diskUsage]
minFreeSpace = <num>

For more details, check here:
http://docs.splunk.com/Documentation/Splunk/latest/Indexer/Setlimitsondiskusage

This can be changed on any Splunk installations as explained on the online documentation:

for all installations, including forwarders, you must have a minimum of 5GB of hard disk space available in addition to the space required for any indexes.

The default is 5000(mb) and this value can be changed as explained before.

For more details, check here:
http://docs.splunk.com/Documentation/Splunk/latest/Installation/Systemrequirements#Recommended_hardw...

woodcock
Esteemed Legend

Go to Settings -> Monitoring Console -> Health Checks and run all the checks. Drill into the red ones and research the details.

0 Karma

rwrettig
New Member

Thank you for replying!

Ran a Health Check.

4 parts are in yellow

  1. Assessment of server ulimits - One or more Splunk instances are running on a host that has one or more resource limits set below official recommendations.
  2. Linux kernel transparent huge pages - One or more Splunk instances are running on a host that has kernel transparent huge pages enabled. This can significantly reduce performance and is against best practice.
  3. Search scheduler skip ratio - Scheduled searches are being skipped on one or more search heads.
  4. System hardware provisioning assessment - One or more hosts has returned CPU or memory specifications that fall below reference hardware recommendations. This might adversely affect indexing or search performance.

I only have one instance running on AWS.

0 Karma

woodcock
Esteemed Legend

Is this a lab? Are you trying to learn or get something hardened for production?

0 Karma

rwrettig
New Member

Learning Splunk!

0 Karma
Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

Splunk is officially part of Cisco

Revolutionizing how our customers build resilience across their entire digital footprint.   Splunk ...

Splunk APM & RUM | Planned Maintenance March 26 - March 28, 2024

There will be planned maintenance for Splunk APM and RUM between March 26, 2024 and March 28, 2024 as ...