All Apps and Add-ons

Do I need to deploy TA-nmon, or can I just pull the NMON files back to Splunk where the NMON Performance Monitor for Unix and LInux Systems is running?

mlapilusa
Engager

We have a local nmon process running on our Linux/AIX servers that generates nmon files on a local directory. We just want to pull the nmon files back to the Splunk search head where the nmon app installed so it can convert them and use them for the nmon app. Is this possible. We don't want to have two apps running nmon on each host. Today I have deployed the TA-nmon app to the servers and this is what I see running after the deployment:

[root@edhlsblap501 rhel]# ps -ef | grep nmon
splunk     333     1  0 08:58 ?        00:00:01 /opt/splunkforwarder/var/run/nmon/bin/linux/rhel/nmon_x86_64_rhel6 -f -T -d 1500 -s 60 -c 120 -p
root     17360     1  0 06:00 ?        00:00:05 /usr/local/bin/nmon -fTN -I 0.001 -s 60 -c 720 -m /perfmetrics/nmon
splunk   23829  6363  4 10:07 ?        00:00:02 /bin/sh /opt/splunkforwarder/etc/apps/TA-nmon/bin/nmon2csv.sh --mode realtime
root     24054  3402  0 10:08 pts/6    00:00:00 grep nmon
[root@edhlsblap501 rhel]#
0 Karma
1 Solution

guilmxm
SplunkTrust
SplunkTrust

Hi,

Please have a look on the online wiki:

http://nmonsplunk.wikidot.com/

Specially read deployment scenarios.

Technically, any instance running either the nmon core app (search head or standalone), the PA-nmon (clustered or standalone indexers) or the TA-nmon (heavy or universal forwarders) can manage nmon data coming from the outside, the is the cold mode scenario.

Therefore, the best (from far) option is having the TA-nmon deployed on your client servers because this provides real monitoring out if the box.
If you have already nmon processes running on servers, I don't recommend trying to use this instead of letting the TA-nmon generates and manage its own data, specially because the TA-nmon runs short time range nmon to avoid CPU overhead, while traditionally people will generate longs time range nmon run. (24 hours)

Nmon is a very small process with a low CPU footprint, having multiple nmon process running is not a problem, it is like having 2 people using top or vmstat command to look at the system load, not a source of CPU overhead.

So, unless you have a real reason to do it (like not having a universal forwarder deployment in your clients) I recommend you the standard TA-nmon as your deployment scenario, advantages are numerous.

Hope this helps !

Guilhem

View solution in original post

guilmxm
SplunkTrust
SplunkTrust

Hi,

Please have a look on the online wiki:

http://nmonsplunk.wikidot.com/

Specially read deployment scenarios.

Technically, any instance running either the nmon core app (search head or standalone), the PA-nmon (clustered or standalone indexers) or the TA-nmon (heavy or universal forwarders) can manage nmon data coming from the outside, the is the cold mode scenario.

Therefore, the best (from far) option is having the TA-nmon deployed on your client servers because this provides real monitoring out if the box.
If you have already nmon processes running on servers, I don't recommend trying to use this instead of letting the TA-nmon generates and manage its own data, specially because the TA-nmon runs short time range nmon to avoid CPU overhead, while traditionally people will generate longs time range nmon run. (24 hours)

Nmon is a very small process with a low CPU footprint, having multiple nmon process running is not a problem, it is like having 2 people using top or vmstat command to look at the system load, not a source of CPU overhead.

So, unless you have a real reason to do it (like not having a universal forwarder deployment in your clients) I recommend you the standard TA-nmon as your deployment scenario, advantages are numerous.

Hope this helps !

Guilhem

michaeljorgense
Path Finder

Does this answer mean I can't use this app, to capture real time performance data from nmon files generated OUTSIDE of TA-nmon?

i.e. I have an AIX server generating nmon data 24 hours a day. I have a Universal Forwarder installed with TA-nmon deployed. Can't I just have TA-nmon, in real time, monitor the AIX generated nmon files & send the data to Splunk?

0 Karma

guilmxm
SplunkTrust
SplunkTrust

Hello Michael,

That is correct.
Technically you can ask the TA-nmon to monitor any nmon file any where, updated in real time or not.

But because running the 24 hours day the nmon file will grow, the processing cost will grow too.
At a certain point, it becomes too high to be reasonable for the cost you can accept from a monitoring tool.

The solution is simple, running 2 nmon process in parallel is not more costly than 2 users looking at topas on the same server, to be honest no big deal no?
Just let the TA-nmon as it is generating its own data, and you'll be good.

You can off course manage numerous things on the TA-nmon and the way it generates its own data, this is all exposed in doc.

Hope this helps.

Regards,

Guilhem

0 Karma

michaeljorgense
Path Finder

Thanks for the reply Guilhem.

It sounds like it is actually possible to monitor nmon files elsewhere in real-time, just not recommended.

How would I go about doing so if I wanted to see what the cost of doing so is like? I assume I'd have to modify a local copy of inputs.conf to point at my separate nmon files.

Would I have to modify any other config files?
Would simply modifying inputs.conf automatically call your parsers and helper scripts when changes were made to my outside TA-nmon nmon files?
Would modifying inputs.conf, automatically stop TA-nmon from generating it's own files or would I have to disable that somewhere also?

I'm quite keen to keep my nmon files as the source of data & monitor them in real-time. So any pointers on how to achieve that would be greatly appreciated 🙂

0 Karma

mlapilusa
Engager

Ok. Thank you Guilhem for detailed reply. That makes sense. But we are seeing the nmon2csv.sh script take up a significant amount of CPU overhead. See below for a capture of the top process on a Linux server where nmon2csv.sh is taking up 5% of the CPU. We are seeing this across all the servers we have deployed the TA-nmon application. Is this normal or can this be optimized to be a smaller CPU footprint? Just FYI, I think this app is great and we love the way we can now gather nmon statistics centrally to provide reports and charts. I just want to make sure we are running it as efficiently as possible.

Top Processes Procs=204 mode=3 (1=Basic, 3=Perf 4=Size 5=I/O)qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqx
x PID %CPU Size Res Res Res Res Shared Faults Command x
x Used KB Set Text Data Lib KB Min Maj x
x 11142 5.5 106612 1348 848 800 0 1136 0 0 nmon2csv.sh

0 Karma

guilmxm
SplunkTrust
SplunkTrust

You're welcome.
Right, it is not taking 5% of the global CPU, actually it is taking 5% of a core (which is what shows the top command), unless you only have one logical CPU in this machine, it is not exactly the same 🙂
Actually, on Linux systems the CPU footprint is normally quote low...

It is possible to lower down the CPU footprint, the first thing you can do is increasing the interval value, take a look here:

http://nmonsplunk.wikidot.com/documentation:userguide:configuration-files:nmonconf

By default, the nmon process runs with a 1 minute interval value, which means 1 minutes between measures. (and so a 1 minute accuracy)
You can increase the value to limit the volume of nmon data, and limit the CPU overhead. (people often do between 1 and 4 minutes interval)

If you decide to increase the interval value, you should also customize the custom span macro:

http://nmonsplunk.wikidot.com/documentation:userguide:configure:coreapp:customspan

Next, you can also enforce the use of the Python parser instead of letting the TA-nmon choose for you (the nmon2csv.sh is a shell wrapper that will decide to use the Perl of Python parser depending on locally available interpreter.)
If you can ensure that all of your linux servers have the minimal requirement (Python 2.7.x available), direclty using the Python parser should have a lower CPU print.
This will be done by deploying a custom "local/props.conf" to TA-nmon deployed to your clients, have a look here:

http://nmonsplunk.wikidot.com/documentation:userguide:configuration-files:propsconf

Note that you can run in trouble if you have Linux servers not respecting the Python requirement.

You can also choose to use the Perl parser instead of the Python parser, if you take a look at nmon_processing data (or the TA-nmon reporting within the app), you will see which parsers are currently being used in your deployment.

Cheers,

Guilhem

0 Karma

mlapilusa
Engager

Perfect. Thank you very much.

0 Karma

guilmxm
SplunkTrust
SplunkTrust

Great, don't hesitate to revert if you have any trouble, and don't forget the rate the app if you like it 🙂

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...