Splunk is causing (Resource temporarily unavailable) on the systems i am monitoring

[robertlynch2020]

Splunk is causing the following error on the application we are running - (Resource temporarily unavailable)

I have run a ulimit -a and i cant see anything wrong - any help wold be great as we have to stop Splunk monitoring till we can get this fixed!!

       at java.lang.ProcessBuilder.start(ProcessBuilder.java:1048)
       at murex.middleware.system.MxProcess.start(MxProcess.java:104)
       ... 18 more
Caused by: java.io.FileNotFoundException: logs/mx_output.log (Resource temporarily unavailable)
       at java.io.FileOutputStream.open0(Native Method)
       at java.io.FileOutputStream.open(FileOutputStream.java:270)
       at java.io.FileOutputStream.<init>(FileOutputStream.java:213)
       at java.lang.ProcessImpl.start(ProcessImpl.java:115)
       at java.lang.ProcessBuilder.start(ProcessBuilder.java:1029)
       ... 19 more

dell427srv autoengine ~/
bash$ ulimit -a
core file size          (blocks, -c) unlimited
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 1546799
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 790527
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 1024
cpu time               (seconds, -t) unlimited
max user processes              (-u) 790527
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited
dell427srv autoengine ~/
bash$ ulimit -a

[nickhills · Accepted solution]

Do you know what the 'mx_output.log' is for? - What process is crashing?
The only thing I can think is that whatever is crashing is demanding exclusive access to the log file, and does not handle it if it cant get a 'lock' while Splunk is reading it. - Very strange.

[robertlynch2020]

Hi

I have new information on this file, it it created and re created at a high rate.

This could be causing the issue right? But splunk should be able to handle it and not cause issues on the service we are monitoring.

In the end i have added it to a black list as we dont need the file

Thanks
Rob

[nickhills]

If Splunk is reading the log when something tries to remove it i guess that could account for the problem.
I wouldn't say its a problem with Splunk - it has to be able to read the file if you want the data indexed, so Splunk is doing what it should - the fact that the other process dumps an exception says to me that it could do with some error/handling retry logic.

If you don't need the contents of the file in Splunk then blacklisting it is the way to go!

[robertlynch2020]

ok cheers for your help

[nickhills]

This line:
java.io.FileNotFoundException: logs/mx_output.log
Suggests that you have a file missing which is causing the script or process to fail.

Can you provide any more detail about that process - what is throwing that exception?

(its not a resource exhaustion/limit)

[robertlynch2020]

When we stop Splunk forwarder it works, when we restart Splunk forwarders we get this error.
To me the machine is low is something, but i dont know what.