why ulimit values are not persistent even after trying all the splunk recommended options

[hegderm]

Hello All,

I have tried updating ulimits values but it is not persistant in all the instances. Production environment which I am working on is having cluster/licensce master, search head cluster, indexer cluster and few heavy forwarder instances.

- As per splunk recommendation tried updating ulimits in /etc/security/limits.conf file and done the service restart. Post that ulimits got persistently updated only in indexer instances and in rest of the instances there was no changes in ulimits value.

- Tried hard coding ulimits values in etc/init.d/splunk file inside splunk start () fucntion as well when it boot starts and rebooted the instances. Post that ulimits persistently updated in only indexer and search head instances 

In heavy forwarder and licence master instances even if I try to increasing ulimits after couple of hpours it is getting reduced and can you please suggest ehat can be done to set the ulimits persistantly 

-

[richgalloway]

Have you tried setting ulimits when starting Splunk?  I put this code in my init.d/splunk file.

change_ulimit() {
  ulimit -Hn 65535
  ulimit -Sn 65535
  ulimit -Hu 20480
  ulimit -Su 20480
  ulimit -Hf unlimited
  ulimit -Sf unlimited
}

Then invoke the function when starting Splunk

case "$1" in
start)
   change_ulimit
   splunk_start
   ;;
stop)
   splunk_stop
   ;;
restart)
   change_ulimit
   splunk_restart
   ;;
status)
	splunk_status
	;;
esac

If you use systemd, try these settings in the Splunk service file:

LimitNOFILE = 65535
LimitNPROC = 20480
LimitFSIZE = infinity

[isoutamo]

And if your environment is using selinux then remember run restoreconn after changes!

r. Ismo

updated:

Create file /etc/security/limits.d/21-splunk.conf with content

splunk       soft    nofile     65536
splunk       hard    nofile     65536

Add also other user specific limits to this file if/when needed.

Then run:

restorecon -FvvR /etc/security/limits.d

 After that new login for user splunk and then it should works.

r. Ismo

[thambisetty]

And it requires reboot of the server also.

[isoutamo]

Actual not, jus logout and login for that user which limit you have changed.

r. Ismo

[thambisetty]

/etc/init.d/splunk is executed at server booting. So I don’t think it works the way you think. 

[isoutamo]

You could run it by hand any time you want. No need to wait a reboot.