Monitoring Splunk

Health Status of Splunkd: Why does the Disk Space show RED?

rwrettig
New Member

On the top of my screen is a red ! - after clicking on it it tells me my disk space is in the red
! Disk Space
-Root Cause
--The diskspace remaining =5434 has breached the red threshold for filesystems=(opt/splunk/var/splunk/audit/db) )
--Last 50 related messages

So I'm out of disk space I presume, so how do i fix it? I have a free setup (I'm a student) with Splunk and an instance on AWS.

Please help!

0 Karma

woodcock
Esteemed Legend

Limits for controlling disk space in Splunk can be changed

The relevant stanza and parameter of interest in server.conf is:

[diskUsage]
minFreeSpace = <num>

For more details, check here:
http://docs.splunk.com/Documentation/Splunk/latest/Indexer/Setlimitsondiskusage

This can be changed on any Splunk installations as explained on the online documentation:

for all installations, including forwarders, you must have a minimum of 5GB of hard disk space available in addition to the space required for any indexes.

The default is 5000(mb) and this value can be changed as explained before.

For more details, check here:
http://docs.splunk.com/Documentation/Splunk/latest/Installation/Systemrequirements#Recommended_hardw...

woodcock
Esteemed Legend

Go to Settings -> Monitoring Console -> Health Checks and run all the checks. Drill into the red ones and research the details.

0 Karma

rwrettig
New Member

Thank you for replying!

Ran a Health Check.

4 parts are in yellow

  1. Assessment of server ulimits - One or more Splunk instances are running on a host that has one or more resource limits set below official recommendations.
  2. Linux kernel transparent huge pages - One or more Splunk instances are running on a host that has kernel transparent huge pages enabled. This can significantly reduce performance and is against best practice.
  3. Search scheduler skip ratio - Scheduled searches are being skipped on one or more search heads.
  4. System hardware provisioning assessment - One or more hosts has returned CPU or memory specifications that fall below reference hardware recommendations. This might adversely affect indexing or search performance.

I only have one instance running on AWS.

0 Karma

woodcock
Esteemed Legend

Is this a lab? Are you trying to learn or get something hardened for production?

0 Karma

rwrettig
New Member

Learning Splunk!

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...