I have TA-nmon - Technical Addon for Nmon Performance Monitor installed on a AIX 6.1 VIO server. I am getting no data to my indexer. All of the folders under /opt/splunkforwarder/var/log/nmon/ are empty or have files with only a 10 characters of data in them.
I have tried the troubleshooting steps listed on the doc site with no luck. I have also tried launching the scripts manually. I get no errors in splunkd as well.
The fifo processes are running...
root 5308504 1 0 14:22:16 pts/1 0:00 /usr/bin/perl /opt/splunkforwarder/etc/apps/TA-nmon/bin/fifo_reader.pl --fifo fifo1
root 9961594 11141350 0 14:54:50 - 0:00 /bin/sh /opt/splunkforwarder/etc/apps/TA-nmon/bin/fifo_reader.sh
/opt/splunkforwarder/var/log/nmon/var/nmon_repository/fifo2/nmon.fifo
root 11141350 1 0 14:54:50 - 0:00 /usr/bin/perl /opt/splunkforwarder/etc/apps/TA-nmon/bin/fifo_reader.pl --fifo fifo2
root 11206756 5308504 0 14:22:16 pts/1 0:00 /bin/sh /opt/splunkforwarder/etc/apps/TA-nmon/bin/fifo_reader.sh
/opt/splunkforwarder/var/log/nmon/var/nmon_repository/fifo1/nmon.fifo
Help is appreciated!
Hi !
Right, the TA-nmon 1.3.x (new fifo implementation) has been qualified against AIX 6.1, so this should definitively work.
Does the topas_nmon process starts as well ?
Seeing the 2 fifo reader processes at the same time should occur only during the parallel run (once a day during a few minutes), so I would think that the topas_nmon process can't start for some reason and only the fifo reader gets started.
Could you please:
Run the script using Splunk interpreter:
/opt/splunkforwarder/bin/splunk cmd /opt/splunkforwarder/etc/apps/TA-nmon/bin/nmon_helper.sh
And share the output with me ? (You can contact me by email on the app main page)
Thank you
Guilhem
Hi !
Right, the TA-nmon 1.3.x (new fifo implementation) has been qualified against AIX 6.1, so this should definitively work.
Does the topas_nmon process starts as well ?
Seeing the 2 fifo reader processes at the same time should occur only during the parallel run (once a day during a few minutes), so I would think that the topas_nmon process can't start for some reason and only the fifo reader gets started.
Could you please:
Run the script using Splunk interpreter:
/opt/splunkforwarder/bin/splunk cmd /opt/splunkforwarder/etc/apps/TA-nmon/bin/nmon_helper.sh
And share the output with me ? (You can contact me by email on the app main page)
Thank you
Guilhem
topas-nmon is not running. So that would make since why there is no data...
i could not find "# set -o vi", i do see a "# set -x" which i commented out to see if it would have an effect.
output from launching nmon_helper with splunk interpreter:
# /opt/splunkforwarder/bin/splunk cmd /opt/splunkforwarder/etc/apps/TA-nmon/bin/nmon_helper.sh
Wed Apr 26 16:04:38 CDT 2017, trdevaix2 INFO: Removing stale pid file
Wed Apr 26 16:04:38 CDT 2017, trdevaix2 INFO: The fifo_reader fifo1 is running
Wed Apr 26 16:04:38 CDT 2017, trdevaix2 INFO: starting the fifo_reader fifo2
Wed Apr 26 16:04:39 CDT 2017, trdevaix2 INFO: starting nmon : /usr/bin/topas_nmon -F /opt/splunkforwarder/var/log/nmon/var/nmon_repository/fifo2/nmon.fifo -T -A -d -K -L -M -P -^ -p -yoverwrite=1 -s 60 -c 1440 in /opt/splunkforwarder/var/log/nmon/var/nmon_repository/fifo2
#
So it looks like the script is starting it to run, but nothing happens, it never shows up under ps aux.
Ensure no process are running, ideally stop the UF to prevent it from starting anything (or disable the input script if you are comfortable with Splunk inputs management)
Can you run manually topas_nmon in a terminal:
/usr/bin/topas_nmon -F /opt/splunkforwarder/var/log/nmon/var/nmon_repository/fifo1/nmon.fifo -T -A -d -K -L -M -P -^ -p -yoverwrite=1 -s 60 -c 1440
In a different terminal run:
cat /opt/splunkforwarder/var/log/nmon/var/nmon_repository/fifo1/nmon.fifo
The expected behavior is that topas_nmon does give you back access to the terminal until you open a reader process (here the cat process)
I suspect the topas_version might be too old and for some version does not support one of the options, or maybe the fifo option (-F)
As well, you can try running:
/usr/bin/topas_nmon -f -T -A -d -K -L -M -P -^ -p -yoverwrite=1 -s 60 -c 1440
And see what happens, this should run nmon in the normal file mode, return you access to the terminal, runs an nmon process in background and write nmon data into an nmon file within the current directory
Looks like you nailed it. Must be an older version of topas nmon 😞
I get this error when trying to run the nmon command manually:
ERROR Invalid option "y"
Hint: topas_nmon [-h] [-s <seconds>] [-c <count>] [-f -d -t -r <name>] [-x]
Command: TOPAS_NMON
-h FULL help information - much more than here
Interactive-Mode:
read startup banner and type: "h" once it is running
For Data-Collect-Mode (-f)
-f spreadsheet output format [note: default -s300 -c288]
optional
-s <seconds> between refreshing the screen [default 2]
-c <number> of refreshes [default millions]
-t spreadsheet includes top processes
-x capacity planning (15 min for 1 day = -fdt -s 900 -c 96)
What is the oldest compatible version of topas-nmon? oslevel currently returns:
# oslevel -s
6100-06-00-0000
Nice 😉
So we're close.
mkdir /opt/splunkforwarder/etc/apps/TA-nmon/local
cp -p /opt/splunkforwarder/etc/apps/TA-nmon/default/nmon.conf /opt/splunkforwarder/etc/apps/TA-nmon/local/
And check if the process has been started.
If so, you can start the UF and the data should start to be managed.
The -yoverwrite=1 option is not that recent, but maybe not that old neither.
And it's mandatory for topas-nmon (At least with last TL)
Remove the option worked! For anyone else, versions prior to TL 9 requires a slightly different switch for fifo write:
-F
Hi David,
Happy to read that 😉
I will check if I can include something dynamic to manage that, maybe possible.
Cheers,
Guilhem
FYI, the TA-nmon 1.3.x has been qualified against AIX 6.1.9.101 (TL09)