All Apps and Add-ons

TA-nmon - Technical Addon for Nmon Performance Monitor: Why am I not receiving any data on AIX VIO server?

david_rose
Communicator

I have TA-nmon - Technical Addon for Nmon Performance Monitor installed on a AIX 6.1 VIO server. I am getting no data to my indexer. All of the folders under /opt/splunkforwarder/var/log/nmon/ are empty or have files with only a 10 characters of data in them.

I have tried the troubleshooting steps listed on the doc site with no luck. I have also tried launching the scripts manually. I get no errors in splunkd as well.

The fifo processes are running...

root  5308504        1   0 14:22:16  pts/1  0:00 /usr/bin/perl /opt/splunkforwarder/etc/apps/TA-nmon/bin/fifo_reader.pl --fifo fifo1

root  9961594 11141350   0 14:54:50      -  0:00 /bin/sh /opt/splunkforwarder/etc/apps/TA-nmon/bin/fifo_reader.sh 
/opt/splunkforwarder/var/log/nmon/var/nmon_repository/fifo2/nmon.fifo

root 11141350        1   0 14:54:50      -  0:00 /usr/bin/perl /opt/splunkforwarder/etc/apps/TA-nmon/bin/fifo_reader.pl --fifo fifo2

root 11206756  5308504   0 14:22:16  pts/1  0:00 /bin/sh /opt/splunkforwarder/etc/apps/TA-nmon/bin/fifo_reader.sh 
/opt/splunkforwarder/var/log/nmon/var/nmon_repository/fifo1/nmon.fifo

Help is appreciated!

0 Karma
1 Solution

guilmxm
Influencer

Hi !

Right, the TA-nmon 1.3.x (new fifo implementation) has been qualified against AIX 6.1, so this should definitively work.

Does the topas_nmon process starts as well ?
Seeing the 2 fifo reader processes at the same time should occur only during the parallel run (once a day during a few minutes), so I would think that the topas_nmon process can't start for some reason and only the fifo reader gets started.

Could you please:

  • Check if you have running topas_nmon process
  • Stop the UF
  • Kill all remaining processes (fifo_reader, etc...)
  • Edit the nmon_helper.sh script and comment out the "# set -o vi" line at the begining
  • Run the script using Splunk interpreter:

    /opt/splunkforwarder/bin/splunk cmd /opt/splunkforwarder/etc/apps/TA-nmon/bin/nmon_helper.sh

And share the output with me ? (You can contact me by email on the app main page)

Thank you

Guilhem

View solution in original post

0 Karma

guilmxm
Influencer

Hi !

Right, the TA-nmon 1.3.x (new fifo implementation) has been qualified against AIX 6.1, so this should definitively work.

Does the topas_nmon process starts as well ?
Seeing the 2 fifo reader processes at the same time should occur only during the parallel run (once a day during a few minutes), so I would think that the topas_nmon process can't start for some reason and only the fifo reader gets started.

Could you please:

  • Check if you have running topas_nmon process
  • Stop the UF
  • Kill all remaining processes (fifo_reader, etc...)
  • Edit the nmon_helper.sh script and comment out the "# set -o vi" line at the begining
  • Run the script using Splunk interpreter:

    /opt/splunkforwarder/bin/splunk cmd /opt/splunkforwarder/etc/apps/TA-nmon/bin/nmon_helper.sh

And share the output with me ? (You can contact me by email on the app main page)

Thank you

Guilhem

0 Karma

david_rose
Communicator

topas-nmon is not running. So that would make since why there is no data...

i could not find "# set -o vi", i do see a "# set -x" which i commented out to see if it would have an effect.

output from launching nmon_helper with splunk interpreter:

# /opt/splunkforwarder/bin/splunk cmd /opt/splunkforwarder/etc/apps/TA-nmon/bin/nmon_helper.sh
Wed Apr 26 16:04:38 CDT 2017, trdevaix2 INFO: Removing stale pid file
Wed Apr 26 16:04:38 CDT 2017, trdevaix2 INFO: The fifo_reader fifo1 is running
Wed Apr 26 16:04:38 CDT 2017, trdevaix2 INFO: starting the fifo_reader fifo2
Wed Apr 26 16:04:39 CDT 2017, trdevaix2 INFO: starting nmon : /usr/bin/topas_nmon -F /opt/splunkforwarder/var/log/nmon/var/nmon_repository/fifo2/nmon.fifo -T -A -d -K -L -M -P -^ -p -yoverwrite=1 -s 60 -c 1440 in /opt/splunkforwarder/var/log/nmon/var/nmon_repository/fifo2
#

So it looks like the script is starting it to run, but nothing happens, it never shows up under ps aux.

0 Karma

guilmxm
Influencer

Ensure no process are running, ideally stop the UF to prevent it from starting anything (or disable the input script if you are comfortable with Splunk inputs management)

Can you run manually topas_nmon in a terminal:

/usr/bin/topas_nmon -F /opt/splunkforwarder/var/log/nmon/var/nmon_repository/fifo1/nmon.fifo -T -A -d -K -L -M -P -^ -p -yoverwrite=1 -s 60 -c 1440

In a different terminal run:

cat /opt/splunkforwarder/var/log/nmon/var/nmon_repository/fifo1/nmon.fifo

The expected behavior is that topas_nmon does give you back access to the terminal until you open a reader process (here the cat process)

I suspect the topas_version might be too old and for some version does not support one of the options, or maybe the fifo option (-F)

As well, you can try running:

/usr/bin/topas_nmon -f -T -A -d -K -L -M -P -^ -p -yoverwrite=1 -s 60 -c 1440

And see what happens, this should run nmon in the normal file mode, return you access to the terminal, runs an nmon process in background and write nmon data into an nmon file within the current directory

0 Karma

david_rose
Communicator

Looks like you nailed it. Must be an older version of topas nmon 😞

I get this error when trying to run the nmon command manually:

ERROR Invalid option "y"

Hint: topas_nmon [-h] [-s <seconds>] [-c <count>] [-f -d -t -r <name>] [-x]
 Command: TOPAS_NMON
        -h            FULL help information - much more than here
        Interactive-Mode:
        read startup banner and type: "h" once it is running
        For Data-Collect-Mode (-f)
        -f            spreadsheet output format [note: default -s300 -c288]
        optional
        -s <seconds>  between refreshing the screen [default 2]
        -c <number>   of refreshes [default millions]
        -t            spreadsheet includes top processes
        -x            capacity planning (15 min for 1 day = -fdt -s 900 -c 96)

What is the oldest compatible version of topas-nmon? oslevel currently returns:

# oslevel -s
6100-06-00-0000
0 Karma

guilmxm
Influencer

Nice 😉

So we're close.

  • Reset the bin/nmon_helper.sh to its origin state (comment out the set -o vi)
  • Create a local nmon.conf:

mkdir /opt/splunkforwarder/etc/apps/TA-nmon/local
cp -p /opt/splunkforwarder/etc/apps/TA-nmon/default/nmon.conf /opt/splunkforwarder/etc/apps/TA-nmon/local/

  • Edit this local/nmon.conf and change the AIX options in "AIX_options" and remove this non compatible option
  • Ensure no processes are running (specially the fifo)
  • Run the nmon_helper.sh by the interpreter

And check if the process has been started.
If so, you can start the UF and the data should start to be managed.

The -yoverwrite=1 option is not that recent, but maybe not that old neither.
And it's mandatory for topas-nmon (At least with last TL)

0 Karma

david_rose
Communicator

Remove the option worked! For anyone else, versions prior to TL 9 requires a slightly different switch for fifo write:

-F

0 Karma

guilmxm
Influencer

Hi David,

Happy to read that 😉
I will check if I can include something dynamic to manage that, maybe possible.

Cheers,

Guilhem

0 Karma

guilmxm
Influencer

FYI, the TA-nmon 1.3.x has been qualified against AIX 6.1.9.101 (TL09)

0 Karma
Get Updates on the Splunk Community!

Last Chance to Submit Your Paper For BSides Splunk - Deadline is August 12th!

Hello everyone! Don't wait to submit - The deadline is August 12th! We have truly missed the community so ...

Ready, Set, SOAR: How Utility Apps Can Up Level Your Playbooks!

 WATCH NOW Powering your capabilities has never been so easy with ready-made Splunk® SOAR Utility Apps. Parse ...

DevSecOps: Why You Should Care and How To Get Started

 WATCH NOW In this Tech Talk we will talk about what people mean by DevSecOps and deep dive into the different ...