Deployment Architecture

Why are my ulimits settings not being respected on ubuntu/debian after a server reboot?

Contributor

I am running Ubuntu version 14 with Splunk version 6.2.2 and have noticed some strangeness with the Ulimits not being respected when the server is rebooted and Splunk is restarted by the OS.

The following edits were made to the /etc/security/limits.conf file:

splunk    hard   nofile   20240
splunk    soft    nofile   10240

The /etc/pam.d/sudo and /etc/pam.d/su files were also edited to add:

session  required  pam_limits.so

Upon starting Splunk manually (i.e. running splunk start from the CLI), the splunkd logs clearly show a limits setting of 20240. However, after a reboot of the server, these limits are not respected and the splunkd logs show an open file limit number of 4096.

Note - I ran the following after installing Splunk to ensure Splunk restarts after a reboot of the server.

splunk enable boot-start
1 Solution

Contributor

It would appear that there is a bug/issue with Ubuntu and perhaps similar Linux OS's where the ulimits settings are not respected, resulting in the above characteristics.

Upon running the splunk enable boot-start command, a Splunk script is added to the /etc/init.d/ directory. A verified fix for this issue involves editing the Splunk script and hard-coding in the hard and soft ulimits to the Splunk script as follows:

  1. Edit: /etc/init.d/splunk
  2. Modify the script so that is looks like the below code segment, adding in the high and low limits
  3. Reboot the server: sudo shutdown -r now

Script Modification:

splunk_start() {
 echo Starting Splunk...
 ulimit -Hn 20240
 ulimit -Sn 10240
 "/opt/splunk/bin/splunk" start --no-prompt --answer-yes
 RETVAL=$?
}

Note: The init.d Splunk script is created only AFTER you have installed Splunk and run the splunk enable boot-start command.

View solution in original post

Path Finder

We ran into this same issue on RHEL 7. After some digging, we found that daemons loaded by systemd ignore anything in /etc/security/limits.conf AND limits.d/*

We wound up extending the service with a file in /etc/systemd/system that sets the limits.

Here are the steps we followed:

First, create directory /etc/systemd/system/splunk.service.d/ if it does not exist.
Then create a file in that directory called "filelimit.conf"

The contents of filelimit.conf should look like this (replacing the limit with whatever value you need):

[Service]
LimitNOFILE=65535

We found this better than trying to modify the init.d script on systems that utilize systemd for startup.

Builder

Thanks @brian.rampley ! Running RHEL 7.2 and indeed the /etc/security/limits.conf was being ignored. My ulimit -a value and what was showing up in splunk (from the health check) were different.

  • created the splunk.service.d directory
  • put the flielimit.conf parameters that weren't matching between my system and splunk health check
  • ran systemctl daemon-reload
  • full system reboot

Did the trick!

0 Karma

Contributor

It should be mentioned that this works (on CentOS 7) after a full host reboot. Just restarting splunk is insufficient.

Engager

By default most modern UNIX systems have pam_limits enabled, and /etc/security/limits.d/

Instead of editing /etc/security/limits.conf, create a file /etc/security/limits.d/99-splunk.com with the following

 splunk    hard   nofile   20240
 splunk    soft    nofile   10240

This just makes it easier to add/remove via any automation (puppet, chef..) or included in your own packages. This will also work if you for some reason chose to run anything splunk as 'root' too 😞 But that is an entirely different problem

(RBAC and facls) are your friend 😉

Contributor

Typo - the last command should have been -
splunk enable boot-start -user splunk

I am running Splunk as the Splunk user, but this issue also appears to affect the root user for those running Splunk as root.

0 Karma

SplunkTrust
SplunkTrust

Hi,

As per your configuration for hardlimit and softlimit, you are setting ulimit for splunk user & you are running splunk as splunk user.

So can you please setup a bootscript with below command if you are running splunk as splunk user

$SPLUNK_HOME/bin/splunk enable boot-start -user splunk

I hope this solves your issue.

Thanks,
Harshil

New Member

Thank you sharing great answers

0 Karma

Contributor

I was running it as a splunk user - just forgot to add that line. Even if you add this - it makes no difference. The issue even affects the root user (if you run splunk as root). The only solution I have found to date is the edit to the script.

0 Karma

Splunk Employee
Splunk Employee

http://docs.splunk.com/Documentation/Splunk/6.6.2/Admin/ConfigureSplunktostartatboottime

Check out this not in docs about updating the init.d for splunk user

0 Karma

SplunkTrust
SplunkTrust

@mcronkrite, Thanks a lot.

That's great to see that the doc has been finally updated and the problem taken in consideration, even if it took ages 😉
But never too late!

0 Karma

Contributor

It would appear that there is a bug/issue with Ubuntu and perhaps similar Linux OS's where the ulimits settings are not respected, resulting in the above characteristics.

Upon running the splunk enable boot-start command, a Splunk script is added to the /etc/init.d/ directory. A verified fix for this issue involves editing the Splunk script and hard-coding in the hard and soft ulimits to the Splunk script as follows:

  1. Edit: /etc/init.d/splunk
  2. Modify the script so that is looks like the below code segment, adding in the high and low limits
  3. Reboot the server: sudo shutdown -r now

Script Modification:

splunk_start() {
 echo Starting Splunk...
 ulimit -Hn 20240
 ulimit -Sn 10240
 "/opt/splunk/bin/splunk" start --no-prompt --answer-yes
 RETVAL=$?
}

Note: The init.d Splunk script is created only AFTER you have installed Splunk and run the splunk enable boot-start command.

View solution in original post

Explorer

thanks for this, fixed our issue by adding this to the splunk_start( ) function

0 Karma

Path Finder

Is this file overwritten every time you upgrade? We are also experiencing this issue and looking for the most practical solution.

0 Karma

SplunkTrust
SplunkTrust

Hi, Splunk does not overwrite the init.d script during upgrade.
It is generated once when you run the "enable boot-start" command during initial installation.

0 Karma
State of Splunk Careers

Access the Splunk Careers Report to see real data that shows how Splunk mastery increases your value and job satisfaction.

Find out what your skills are worth!