topic When Linux opens files, splunk crashes due to file descriptors-How can I fix this? in Monitoring Splunk

When Linux opens files, splunk crashes due to file descriptors-How can I fix this?

Gabriel_CCI — Mon, 11 Apr 2022 20:31:51 GMT

Dear All.
I have a problem with the number of files opened by Linux, I have set the ulimit parameter to 999999 but I still have Splunk service crashes due to file descriptors, this happens in the Search heads, is there a way to tell Splunk not to open more files ?, I have tried with

[inputproc]
max_fd = 120000

but it keeps opening many more files

The Linux version is

Oracle Linux Server release 7.8

Re: When Linux opens files, splunk crashes due to file descriptors-How can I fix this?

gcusello — Tue, 12 Apr 2022 06:34:51 GMT

Hi @Gabriel_CCI,

did you performed the Monitor Console Health Check?

In this way you can be sure about the parameter.

Anyway, you have to configure ulimit for the user running Splunk (usually splunk or root,

in othe words, in "/etc/security/limits.conf" you have to configure (if your Splunk is running under the "splunk" user) :

splunk hard nofile 999999 splunk soft nofile 999999

Ciao.

Giuseppe

Re: When Linux opens files, splunk crashes due to file descriptors-How can I fix this?

Gabriel_CCI — Tue, 12 Apr 2022 12:39:18 GMT

Yes, the parameter 999999 was set for the splunk user, but it still crashes and another symptom that I have detected is that when many files are opened in the SH port 8089 is queued.

Re: When Linux opens files, splunk crashes due to file descriptors-How can I fix this?

gcusello — Tue, 12 Apr 2022 12:41:55 GMT

Hi @Gabriel_CCI,

the smartest solution is to open a Case to Splunk Support.

Did you used Monitor Console Health Check, to be sure that the ulimit is applied?

Ciao.

Giuseppe

Re: When Linux opens files, splunk crashes due to file descriptors-How can I fix this?

PickleRick — Tue, 12 Apr 2022 12:46:05 GMT

With systemd-managed service the limits are defined in the service unit file - /etc/systemd/system/<whatever_you_called_your_service> (by default it's called Splunkd.service_<date>).

There you have whole [Service] section which contains the limits (among other things).

If you edit the file, remember to reload the systemd configuration so it picks up new settings.

Re: When Linux opens files, splunk crashes due to file descriptors-How can I fix this?

Gabriel_CCI — Fri, 29 Apr 2022 13:45:02 GMT

How I can count the number of open files per minute by query in Splunk?

thanks