topic Re: Changing the Ulimits for openfiles in Splunk Search

Changing the Ulimits for openfiles

nawazns5038 — Wed, 17 Jan 2018 18:39:09 GMT

How can we change the ulimits of Splunk to the desired value ?
I have edited the /etc/security/limits.conf file and rebooted the instance
I have added "* - nofile 64000" to the file .
But Splunk still shows only 4096. How can we change this value .

Re: Changing the Ulimits for openfiles

nickhills — Wed, 17 Jan 2018 18:45:50 GMT

Did you set both hard and soft limits?

Take a look at this -

https://www.cyberciti.biz/faq/linux-increase-the-maximum-number-of-open-files/

You may be seeing the soft limit, even though you have raised the system wide hard limit

Re: Changing the Ulimits for openfiles

klaxdal — Wed, 17 Jan 2018 18:48:42 GMT

Try setting it in the Splunk initd
https://answers.splunk.com/answers/223838/why-are-my-ulimits-settings-not-being-respected-on.html

Re: Changing the Ulimits for openfiles

mayurr98 — Wed, 17 Jan 2018 18:51:28 GMT

hey have a look at this
you can create a script!
https://answers.splunk.com/answers/223838/why-are-my-ulimits-settings-not-being-respected-on.html

Also
https://answers.splunk.com/answers/13313/how-to-tune-ulimit-on-my-server.html

https://www.cyberciti.biz/faq/linux-increase-the-maximum-number-of-open-files/

Re: Changing the Ulimits for openfiles

nawazns5038 — Wed, 17 Jan 2018 23:28:42 GMT

Is it necessary to change the ulimts of all the Splunk instances or just the indexers ?

Re: Changing the Ulimits for openfiles

nawazns5038 — Wed, 17 Jan 2018 23:28:48 GMT

Is it necessary to change the ulimts of all the Splunk instances or just the indexers ?

Re: Changing the Ulimits for openfiles

nawazns5038 — Wed, 17 Jan 2018 23:29:07 GMT

Is it necessary to change the ulimts of all the Splunk instances or just the indexers ?

Re: Changing the Ulimits for openfiles

nawazns5038 — Thu, 18 Jan 2018 02:06:21 GMT

Any standard procedure for changing the ulimits of the servers ?
I tried "ulimit -n 65536" command on all the indexers and did a rolling restart on the servers. Still there is no change in the ulimits. and edited the /etc/init.d/splunk script as mentioned.

Is it the right process to be followed ?

Re: Changing the Ulimits for openfiles

nawazns5038 — Thu, 18 Jan 2018 02:06:33 GMT

Is it the right process to be followed ?

Re: Changing the Ulimits for openfiles

nawazns5038 — Thu, 18 Jan 2018 02:06:44 GMT

Is it the right process to be followed ?

Re: Changing the Ulimits for openfiles

mayurr98 — Thu, 18 Jan 2018 06:43:14 GMT

see this splunk official doc
http://docs.splunk.com/Documentation/Splunk/latest/Troubleshooting/ulimitErrors

Re: Changing the Ulimits for openfiles

klaxdal — Fri, 19 Jan 2018 17:59:23 GMT

I personally change the ulimits on all splunk servers ( for consistency )

I also use the initd method to ensure ulimits are 'stickey'

** note you will need to full restart the server ( reboot ) for the changes to take effect as they are in the initd running /opt/splunk/bin/splunk restart wont put the required change in place .

Re: Changing the Ulimits for openfiles

nawazns5038 — Mon, 22 Jan 2018 19:36:06 GMT

how can reboot all the servers of a cluster safely ?
Rebooting is fine if its a single instance

Re: Changing the Ulimits for openfiles

nickhills — Mon, 22 Jan 2018 19:50:15 GMT

No, you will need to restart the OS, not just splunk.
Take a look at maintenance mode
http://docs.splunk.com/Documentation/Splunk/7.0.1/Indexer/Usemaintenancemode

Re: Changing the Ulimits for openfiles

nawazns5038 — Thu, 25 Jan 2018 00:46:18 GMT

@nickhillscpl , how can we restart the OS without effecting Splunk service.
Maintenance mode only restarts the Splunk service.

Re: Changing the Ulimits for openfiles

nickhills — Thu, 25 Jan 2018 07:11:47 GMT

Maintenance mode is useful if you are restarting a cluster peer. If the machine does not restart within the cluster timeout, a fix up operation will begin on all missing replicas. You want this to happen if one of your peers fails, but not if your rebooting.

Enabling maint. Mode prevents the cluster performing any fixup operations while you restart your servers.

To perform a clean os restart, enable maintenance mode on your cluster master, then run ‘splunk offline’ on a peer, and restart the os. When that peer is back up and connected to the cluster, repeat the process for other peers.
Then disable maint. Mode