topic why ulimit values are not persistent even after trying all the splunk recommended options in Splunk Search

why ulimit values are not persistent even after trying all the splunk recommended options

hegderm — Fri, 14 Aug 2020 07:01:35 GMT

Hello All,

I have tried updating ulimits values but it is not persistant in all the instances. Production environment which I am working on is having cluster/licensce master, search head cluster, indexer cluster and few heavy forwarder instances.

- As per splunk recommendation tried updating ulimits in /etc/security/limits.conf file and done the service restart. Post that ulimits got persistently updated only in indexer instances and in rest of the instances there was no changes in ulimits value.

- Tried hard coding ulimits values in etc/init.d/splunk file inside splunk start () fucntion as well when it boot starts and rebooted the instances. Post that ulimits persistently updated in only indexer and search head instances

In heavy forwarder and licence master instances even if I try to increasing ulimits after couple of hpours it is getting reduced and can you please suggest ehat can be done to set the ulimits persistantly

Re: why ulimit values are not persistent even after trying all the splunk recommended options

richgalloway — Fri, 14 Aug 2020 13:08:14 GMT

Have you tried setting ulimits when starting Splunk? I put this code in my init.d/splunk file.

change_ulimit() { ulimit -Hn 65535 ulimit -Sn 65535 ulimit -Hu 20480 ulimit -Su 20480 ulimit -Hf unlimited ulimit -Sf unlimited }

Then invoke the function when starting Splunk

case "$1" in start) change_ulimit splunk_start ;; stop) splunk_stop ;; restart) change_ulimit splunk_restart ;; status) splunk_status ;; esac

If you use systemd, try these settings in the Splunk service file:

LimitNOFILE = 65535 LimitNPROC = 20480 LimitFSIZE = infinity

Re: why ulimit values are not persistent even after trying all the splunk recommended options

isoutamo — Mon, 17 Aug 2020 11:20:48 GMT

And if your environment is using selinux then remember run restoreconn after changes!

r. Ismo

updated:

Create file /etc/security/limits.d/21-splunk.conf with content

splunk soft nofile 65536 splunk hard nofile 65536

Add also other user specific limits to this file if/when needed.

Then run:

restorecon -FvvR /etc/security/limits.d

After that new login for user splunk and then it should works.

r. Ismo

Re: why ulimit values are not persistent even after trying all the splunk recommended options

thambisetty — Fri, 14 Aug 2020 19:02:43 GMT

And it requires reboot of the server also.

Re: why ulimit values are not persistent even after trying all the splunk recommended options

isoutamo — Fri, 14 Aug 2020 19:05:12 GMT

Actual not, jus logout and login for that user which limit you have changed.

r. Ismo

Re: why ulimit values are not persistent even after trying all the splunk recommended options

thambisetty — Fri, 14 Aug 2020 19:07:48 GMT

/etc/init.d/splunk is executed at server booting. So I don’t think it works the way you think.

Re: why ulimit values are not persistent even after trying all the splunk recommended options

isoutamo — Fri, 14 Aug 2020 19:09:25 GMT

You could run it by hand any time you want. No need to wait a reboot.