topic Re: Why are my ulimits settings not being respected on ubuntu/debian after a server reboot? in Deployment Architecture

Why are my ulimits settings not being respected on ubuntu/debian after a server reboot?

pj — Wed, 25 Mar 2015 19:11:35 GMT

I am running Ubuntu version 14 with Splunk version 6.2.2 and have noticed some strangeness with the Ulimits not being respected when the server is rebooted and Splunk is restarted by the OS.

The following edits were made to the /etc/security/limits.conf file:

splunk    hard   nofile   20240
splunk    soft    nofile   10240

The /etc/pam.d/sudo and /etc/pam.d/su files were also edited to add:

session  required  pam_limits.so

Upon starting Splunk manually (i.e. running splunk start from the CLI), the splunkd logs clearly show a limits setting of 20240. However, after a reboot of the server, these limits are not respected and the splunkd logs show an open file limit number of 4096.

Note - I ran the following after installing Splunk to ensure Splunk restarts after a reboot of the server.

splunk enable boot-start

Re: Why are my ulimits settings not being respected on ubuntu/debian after a server reboot?

pj — Wed, 25 Mar 2015 19:22:35 GMT

It would appear that there is a bug/issue with Ubuntu and perhaps similar Linux OS's where the ulimits settings are not respected, resulting in the above characteristics.

Upon running the splunk enable boot-start command, a Splunk script is added to the /etc/init.d/ directory. A verified fix for this issue involves editing the Splunk script and hard-coding in the hard and soft ulimits to the Splunk script as follows:

Edit: /etc/init.d/splunk
Modify the script so that is looks like the below code segment, adding in the high and low limits
Reboot the server: sudo shutdown -r now

Script Modification:

splunk_start() {
 echo Starting Splunk...
 ulimit -Hn 20240
 ulimit -Sn 10240
 "/opt/splunk/bin/splunk" start --no-prompt --answer-yes
 RETVAL=$?
}

Note: The init.d Splunk script is created only AFTER you have installed Splunk and run the splunk enable boot-start command.

Re: Why are my ulimits settings not being respected on ubuntu/debian after a server reboot?

harsmarvania57 — Wed, 25 Mar 2015 19:38:40 GMT

Hi,

As per your configuration for hardlimit and softlimit, you are setting ulimit for splunk user & you are running splunk as splunk user.

So can you please setup a bootscript with below command if you are running splunk as splunk user

$SPLUNK_HOME/bin/splunk enable boot-start -user splunk

I hope this solves your issue.

Thanks,
Harshil

Re: Why are my ulimits settings not being respected on ubuntu/debian after a server reboot?

pj — Wed, 25 Mar 2015 19:48:55 GMT

I was running it as a splunk user - just forgot to add that line. Even if you add this - it makes no difference. The issue even affects the root user (if you run splunk as root). The only solution I have found to date is the edit to the script.

Re: Why are my ulimits settings not being respected on ubuntu/debian after a server reboot?

pj — Wed, 25 Mar 2015 19:52:31 GMT

Typo - the last command should have been -
splunk enable boot-start -user splunk

I am running Splunk as the Splunk user, but this issue also appears to affect the root user for those running Splunk as root.

Re: Why are my ulimits settings not being respected on ubuntu/debian after a server reboot?

oxdeadbeef — Wed, 26 Aug 2015 17:54:26 GMT

By default most modern UNIX systems have pam_limits enabled, and /etc/security/limits.d/

Instead of editing /etc/security/limits.conf, create a file /etc/security/limits.d/99-splunk.com with the following

 splunk    hard   nofile   20240
 splunk    soft    nofile   10240

This just makes it easier to add/remove via any automation (puppet, chef..) or included in your own packages. This will also work if you for some reason chose to run anything splunk as 'root' too 😞 But that is an entirely different problem

(RBAC and facls) are your friend 😉

Re: Why are my ulimits settings not being respected on ubuntu/debian after a server reboot?

brian_rampley — Tue, 16 Aug 2016 14:29:05 GMT

We ran into this same issue on RHEL 7. After some digging, we found that daemons loaded by systemd ignore anything in /etc/security/limits.conf AND limits.d/*

We wound up extending the service with a file in /etc/systemd/system that sets the limits.

Here are the steps we followed:

First, create directory /etc/systemd/system/splunk.service.d/ if it does not exist.
Then create a file in that directory called "filelimit.conf"

The contents of filelimit.conf should look like this (replacing the limit with whatever value you need):

[Service]
LimitNOFILE=65535

We found this better than trying to modify the init.d script on systems that utilize systemd for startup.

Re: Why are my ulimits settings not being respected on ubuntu/debian after a server reboot?

robert_miller — Wed, 28 Jun 2017 18:11:32 GMT

Is this file overwritten every time you upgrade? We are also experiencing this issue and looking for the most practical solution.

Re: Why are my ulimits settings not being respected on ubuntu/debian after a server reboot?

guilmxm — Fri, 30 Jun 2017 19:11:53 GMT

Hi, Splunk does not overwrite the init.d script during upgrade.
It is generated once when you run the "enable boot-start" command during initial installation.

Re: Why are my ulimits settings not being respected on ubuntu/debian after a server reboot?

mcronkrite_splu — Sat, 05 Aug 2017 23:01:02 GMT

http://docs.splunk.com/Documentation/Splunk/6.6.2/Admin/ConfigureSplunktostartatboottime

Check out this not in docs about updating the init.d for splunk user

Re: Why are my ulimits settings not being respected on ubuntu/debian after a server reboot?

guilmxm — Wed, 09 Aug 2017 14:26:32 GMT

@mcronkrite, Thanks a lot.

That's great to see that the doc has been finally updated and the problem taken in consideration, even if it took ages 😉
But never too late!

Re: Why are my ulimits settings not being respected on ubuntu/debian after a server reboot?

DUThibault — Tue, 31 Oct 2017 15:03:26 GMT

It should be mentioned that this works (on CentOS 7) after a full host reboot. Just restarting splunk is insufficient.

Re: Why are my ulimits settings not being respected on ubuntu/debian after a server reboot?

DEAD_BEEF — Mon, 18 Dec 2017 15:55:48 GMT

Thanks @brian.rampley ! Running RHEL 7.2 and indeed the /etc/security/limits.conf was being ignored. My ulimit -a value and what was showing up in splunk (from the health check) were different.

created the splunk.service.d directory
put the flielimit.conf parameters that weren't matching between my system and splunk health check
ran systemctl daemon-reload
full system reboot

Did the trick!

Re: Why are my ulimits settings not being respected on ubuntu/debian after a server reboot?

ehorwood — Fri, 23 Feb 2018 16:44:43 GMT

thanks for this, fixed our issue by adding this to the splunk_start( ) function

Re: Why are my ulimits settings not being respected on ubuntu/debian after a server reboot?

manchun — Fri, 23 Feb 2018 16:47:04 GMT

Thank you sharing great answers