Monitoring Splunk

Not enough free disk space on the search head, and not enough free swap space on the indexer

Engager

Hi all,
We enlarged RAM and CPU capabilites on the indexer and search head. After that we had a problem on these servers: the free disk space decreases dramatically on the search head (in a few minutes to 0%), and the free swap space decreases dramatically on the indexer.
After servers reboot the problem temporarily disappears. The problem repeats in a few days.
There was no such problem before enlarging RAM and CPU capabilites.
In the logs splunk did not find anytheng.
Can this problem be related to the enlarging RAM and CPU capabilites? Where else to look for the cause of this problem?
Guys could you comment on this?

We have 1 indexer and several search head:
Splunk Enterprise Server 6.5.2
Linux, 47.1 GB Physical Memory, 12 CPU Cores

Labels (1)

Esteemed Legend

What exactly do you mean by We enlarged RAM and CPU capabilites?

0 Karma

Motivator

Be sure to configure logrotate for the Splunk internal logs, they can grow to around 25MB each pretty quickly.
Another option is to symlink /opt/splunk/var/log/splunk over to /var/log/

Here is how I configure logrotate on my Splunk nodes:

/etc/logrotate.d/splunk
/var/log/splunk/*.log {
    missingok
    notifempty
    compress
    daily
    rotate 7
    create 0644 splunk splunk
}
0 Karma

Contributor

Sorry, but i think, this is not gonna help. @lkeli_spl please comment.

0 Karma
State of Splunk Careers

Access the Splunk Careers Report to see real data that shows how Splunk mastery increases your value and job satisfaction.

Find out what your skills are worth!