Splunk Search

why ulimit values are not persistent even after trying all the splunk recommended options

hegderm
Engager

Hello All,

I have tried updating ulimits values but it is not persistant in all the instances. Production environment which I am working on is having cluster/licensce master, search head cluster, indexer cluster and few heavy forwarder instances.

- As per splunk recommendation tried updating ulimits in /etc/security/limits.conf file and done the service restart. Post that ulimits got persistently updated only in indexer instances and in rest of the instances there was no changes in ulimits value.

- Tried hard coding ulimits values in etc/init.d/splunk file inside splunk start () fucntion as well when it boot starts and rebooted the instances. Post that ulimits persistently updated in only indexer and search head instances 

In heavy forwarder and licence master instances even if I try to increasing ulimits after couple of hpours it is getting reduced and can you please suggest ehat can be done to set the ulimits persistantly 

 

-

Labels (1)
Tags (1)
0 Karma

richgalloway
SplunkTrust
SplunkTrust

Have you tried setting ulimits when starting Splunk?  I put this code in my init.d/splunk file.

change_ulimit() {
  ulimit -Hn 65535
  ulimit -Sn 65535
  ulimit -Hu 20480
  ulimit -Su 20480
  ulimit -Hf unlimited
  ulimit -Sf unlimited
}

Then invoke the function when starting Splunk

case "$1" in
start)
   change_ulimit
   splunk_start
   ;;
stop)
   splunk_stop
   ;;
restart)
   change_ulimit
   splunk_restart
   ;;
status)
	splunk_status
	;;
esac

If you use systemd, try these settings in the Splunk service file:

LimitNOFILE = 65535
LimitNPROC = 20480
LimitFSIZE = infinity
---
If this reply helps you, Karma would be appreciated.

isoutamo
SplunkTrust
SplunkTrust

And if your environment is using selinux then remember run restoreconn after changes!

r. Ismo

updated:

Create file /etc/security/limits.d/21-splunk.conf with content

 

splunk       soft    nofile     65536
splunk       hard    nofile     65536

 

Add also other user specific limits to this file if/when needed.

Then run:

 

restorecon -FvvR /etc/security/limits.d

 

 After that new login for user splunk and then it should works.

r. Ismo

thambisetty
SplunkTrust
SplunkTrust

And it requires reboot of the server also.

————————————
If this helps, give a like below.

isoutamo
SplunkTrust
SplunkTrust

Actual not, jus logout and login for that user which limit you have changed.

r. Ismo

thambisetty
SplunkTrust
SplunkTrust

/etc/init.d/splunk is executed at server booting. So I don’t think it works the way you think. 

————————————
If this helps, give a like below.
0 Karma

isoutamo
SplunkTrust
SplunkTrust

You could run it by hand any time you want. No need to wait a reboot.

Get Updates on the Splunk Community!

Built-in Service Level Objectives Management to Bridge the Gap Between Service & ...

Now On-Demand Join us to learn more about how you can leverage Service Level Objectives (SLOs) and the new ...

Database Performance Sidebar Panel Now on APM Database Query Performance & Service ...

We’ve streamlined the troubleshooting experience for database-related service issues by adding a database ...

IM Landing Page Filter - Now Available

We’ve added the capability for you to filter across the summary details on the main Infrastructure Monitoring ...