NMON Performance Monitor for Unix and Linux Systems: Why is my filesystem filling up with core.### files?

[phswartz]

Hello,

My filesystem is filling up with core.### files in /opt/splunk/var/run/nmon/var/nmon_repository.

What could be causing that and how can I make it quit?

file core.44614
core.44614: ELF 64-bit LSB core file x86-64, version 1 (SYSV), SVR4-style, from '/opt/splunk/var/run/nmon/bin/linux/rhel/nmon_x86_64_rhel6 -f -T -d 1500 -s 60 -'

[guilmxm]

Hello,

You may be lacking some basic system libraries.

If this occurs on a search head or standalone instance:

To deactivate the input script, look in the app bar menu --> "Settings" --> "Nmon local perf collector" and deactivate the script input.

If this occurs on a Universal Forwarder client, uninstall the TA-nmon addon.

Therefore, you can also easily investigate, in terminal run:

/opt/splunk/var/run/nmon/bin/linux/rhel/nmon_x86_64_rhel6

You should get a message with the name of the library lacking.

You can also install the nmon binary for Linux for your OS, and ensure its runs as expected by launching the command "nmon"

If for some reason the embedded binary that should suit your OS does not (this would be strange !), installing the nmon binary in your system will make the input script (nmon_helper.sh) use the nmon bin in priority. (instead of embedded binaries)

Please let me know the outcome.

[phswartz]

Hi... the local version of nmon is 14i and the included version for RHEL6 is 16d. I do have the priority for the local version, but in the app UI for these systems it is reporting the 16d version.
Both versions work w/o error when running from the cmd line.

[guilmxm]

Does this occurs on a Universal Forwarder client ?

Can you try running nmon in output mode:

/opt/splunk/var/run/nmon/bin/linux/rhel/nmon_x86_64_rhel6 -f -T -d 1500 -s 60 -c 120 -p

This should exit, generate a running nmon process and an nmon file in the current directory

[phswartz]

Yes, it is on a UF client.

root@sappr3ap14 # /opt/splunk/var/run/nmon/bin/linux/rhel/nmon_x86_64_rhel6 -f -T -d 1500 -s 60 -c 120 -p
79600

ls -lart
-rw-r--r--.  1 root   root      65585 Apr  4 12:25 sappr3ap14_160404_1224.nmon

I also have the Splunk_TA_nix on the UF if that is playing a part.
Thanks

[guilmxm]

Right.

Please run the trouble shooting procedure:

http://nmonsplunk.wikidot.com/documentation:userguide:troubleshoot:troubleguide

Start at step 2 and try running manually the script "nmon_helper.sh":

/opt/splunkforwarder/bin/splunk cmd /opt/splunkforwarder/etc/apps/TA-nmon/bin/nmon_helper.sh

[phswartz]

Okay... interesting ... on this example client the search index=nmon sourcetype=nmon_collect host=sappr3ap14 comes back empty, (but other clients work).

Then for the ps command ...

root@sappr3ap14 # ps -ef |grep nmon
root      53553      1  0 00:01 ?        00:00:24 /usr/bin/nmon -f -s 120 -c 720 -TN -m /tmp -p
root      86760      1  0 12:57 ?        00:00:00 /opt/splunk/var/run/nmon/bin/linux/rhel/nmon_x86_64_rhel6 -f -T -d 1500 -s 60 -c 120 -p
root      86857  44758  0 12:57 pts/0    00:00:00 grep nmon

So, continued on in the troubleshooting and shutdown splunk, killed all running nmon pid, and ran the nmon_helper.sh.
Restarted splunk and now only the embedded nmon (16d) is seen to be running and I'm not getting the core.### files.

Not sure why the embedded version is being used when the local version is available and has the priority, but guess since the embedded version is much newer it probably doesn't matter.

Thanks for your help... now on to the other 10 that are having this issue.....

[guilmxm]

Right, i will double verify that the priority feature works as expected, but as far i as know it was 🙂

I see in your ps output:

/usr/bin/nmon -f -s 120 -c 720 -TN -m /tmp -p

Is it an nmon process you spawned manually ? Or nmon collections you run on your owns ?

Still it should not generate any trouble, and definitively not generates cores, quite strange

[guilmxm]

Ok, after having double checked, the Linux nmon priority feature works as expected,with:

Linux_embedded_nmon_priority="0"

In local/nmon.conf, the nmon_helper.sh will use any nmon binary available in PATH before trying to use embedded binaries.

I have updated the documentation which was not up to date, the default configuration sets priority to embedded binaries since v1.6.07:

http://nmonsplunk.wikidot.com/documentation:userguide:configure:collection:priority

In your case, the presence of the other nmon process is not related to the TA-nmon (options sent to the nmon exe are not the one used by the TA-nmon)

Still i do know what was causing cores in your deployment...

[guilmxm]

Can you contact me by mail ?

On the app home page:
https://splunkbase.splunk.com/app/1753/

Click on "contact developer"