Monitoring Splunk

Not enough free disk space on the search head, and not enough free swap space on the indexer

lkeli_spl
Engager

Hi all,
We enlarged RAM and CPU capabilites on the indexer and search head. After that we had a problem on these servers: the free disk space decreases dramatically on the search head (in a few minutes to 0%), and the free swap space decreases dramatically on the indexer.
After servers reboot the problem temporarily disappears. The problem repeats in a few days.
There was no such problem before enlarging RAM and CPU capabilites.
In the logs splunk did not find anytheng.
Can this problem be related to the enlarging RAM and CPU capabilites? Where else to look for the cause of this problem?
Guys could you comment on this?

We have 1 indexer and several search head:
Splunk Enterprise Server 6.5.2
Linux, 47.1 GB Physical Memory, 12 CPU Cores

Labels (1)

woodcock
Esteemed Legend

What exactly do you mean by We enlarged RAM and CPU capabilites?

0 Karma

codebuilder
Influencer

Be sure to configure logrotate for the Splunk internal logs, they can grow to around 25MB each pretty quickly.
Another option is to symlink /opt/splunk/var/log/splunk over to /var/log/

Here is how I configure logrotate on my Splunk nodes:

/etc/logrotate.d/splunk
/var/log/splunk/*.log {
    missingok
    notifempty
    compress
    daily
    rotate 7
    create 0644 splunk splunk
}
----
An upvote would be appreciated and Accept Solution if it helps!
0 Karma

highsplunker
Contributor

Sorry, but i think, this is not gonna help. @lkeli_spl please comment.

0 Karma
Get Updates on the Splunk Community!

Celebrating Fast Lane: 2025 Authorized Learning Partner of the Year

At .conf25, Splunk proudly recognized Fast Lane as the 2025 Authorized Learning Partner of the Year. This ...

Tech Talk Recap | Mastering Threat Hunting

Mastering Threat HuntingDive into the world of threat hunting, exploring the key differences between ...

Observability for AI Applications: Troubleshooting Latency

If you’re working with proprietary company data, you’re probably going to have a locally hosted LLM or many ...