All Apps and Add-ons

NMON - VIOS (AIX) -

nkrabbe_splunk
Splunk Employee
Splunk Employee

Hello,

I recently installed the "NMON Performance Monitor for Unix and Linux Systems" app and it works great for NMON output for standard AIX servers. We have ran into issues with AIX VIOS servers. The format of the data looks very similar to the AIX servers that worked correctly. However the nmon2csv.pl script seems like it is not generating the csv file in the csv_repository directory and thus no data is coming in.

When I run a search for the hostname of one of these VIOS servers I see that it did index a single event for it with sourcetype=nmon_config but nothing for sourcetype=nmon_data.

Any thoughts on what may be wrong or ideas of how to go about troubleshooting this? Thanks!

0 Karma
1 Solution

guilmxm
SplunkTrust
SplunkTrust

Hi,

It should work too with AIX Vios servers, i have worked with Nmon data from Vios servers with no Vios specific issues.

Thus a few things could lead the converter to ignore nmon files:

  • ZZZZ lines truncated: this sometimes seems to happen with some buggy nmon versions, a ZZZZ line contains the timestamp of one event measure and is extracted by the script

A line containing the ZZZZ pattern should always begin the line, if the script finds a line containing the ZZZZ pattern but not begin the line, the nmon file will be ignored by the converter to prevent data inconstancy

  • Obsolete nmon version (v9 and before) was generating data trouble and i decided to prevent the script from working on these old format until i terminate some improvement within the script (this is mainly due to different time format of events)

Note: Since Version 1.3.4, these error events can be retrieved in the report "Errors in NMON processing" (this required you to run at least version 1.3.4 when operation were performed), or manually:

index=nmon sourcetype="nmon_processing" error

If you ran in one of these case, you should find an entry for your hostname

Note: you can manually search for data indexed (perf data) with such manual search:

index=nmon sourcetype=nmon_data hostname=XXXXXX | stats count by type

--> Will return the number of events by type of perf monitor for a given host

Otherwise, the script may fall into some unexpected case which leads it to fail converting data, you can manually debug the situation.

On a test indexer environment (if you have some), temporarily deactivate the input monitor which consumes csv data (you'll find it into input within the manager)

Get a copy an nmon file generated in one of you Aix vios servers, and:

Under terminal:

$ export SPLUNK_HOME=/opt/splunk
$ cd etc/apps/nmon/bin
$ cat | ./nmon2csv.pl

And see what happens, any error encountered by the script...
If the operation works as expected, this should indeed generate csv files within the csv_repository directory

i will be interested in any element, alternatively if you can send an nmon file sample from your system, i'll be happy to debug what's going on, and if required process to an App update.
You can contact me through Splunk base or the App page

Let me know

Guilhem

View solution in original post

0 Karma

guilmxm
SplunkTrust
SplunkTrust

After our exchange and analysis, i can confirm this is being caused by blank lines present in these nmon files, which causes the converter to fail generating expected data.

A new Release as of the Version 1.3.6 has been published which corrects this (and brings some other changes and corrections), please update and retry to manage nmon files from the same source, and revert.

Thank for having pointing this out !

Cheers

Guilhem

nkrabbe_splunk
Splunk Employee
Splunk Employee

The latest release has fixed my issue. Thanks so much Guilhem! Great app.

0 Karma

guilmxm
SplunkTrust
SplunkTrust

Hi,

It should work too with AIX Vios servers, i have worked with Nmon data from Vios servers with no Vios specific issues.

Thus a few things could lead the converter to ignore nmon files:

  • ZZZZ lines truncated: this sometimes seems to happen with some buggy nmon versions, a ZZZZ line contains the timestamp of one event measure and is extracted by the script

A line containing the ZZZZ pattern should always begin the line, if the script finds a line containing the ZZZZ pattern but not begin the line, the nmon file will be ignored by the converter to prevent data inconstancy

  • Obsolete nmon version (v9 and before) was generating data trouble and i decided to prevent the script from working on these old format until i terminate some improvement within the script (this is mainly due to different time format of events)

Note: Since Version 1.3.4, these error events can be retrieved in the report "Errors in NMON processing" (this required you to run at least version 1.3.4 when operation were performed), or manually:

index=nmon sourcetype="nmon_processing" error

If you ran in one of these case, you should find an entry for your hostname

Note: you can manually search for data indexed (perf data) with such manual search:

index=nmon sourcetype=nmon_data hostname=XXXXXX | stats count by type

--> Will return the number of events by type of perf monitor for a given host

Otherwise, the script may fall into some unexpected case which leads it to fail converting data, you can manually debug the situation.

On a test indexer environment (if you have some), temporarily deactivate the input monitor which consumes csv data (you'll find it into input within the manager)

Get a copy an nmon file generated in one of you Aix vios servers, and:

Under terminal:

$ export SPLUNK_HOME=/opt/splunk
$ cd etc/apps/nmon/bin
$ cat | ./nmon2csv.pl

And see what happens, any error encountered by the script...
If the operation works as expected, this should indeed generate csv files within the csv_repository directory

i will be interested in any element, alternatively if you can send an nmon file sample from your system, i'll be happy to debug what's going on, and if required process to an App update.
You can contact me through Splunk base or the App page

Let me know

Guilhem

View solution in original post

0 Karma

guilmxm
SplunkTrust
SplunkTrust

I see, ok first i see you're using Mac OS X, that's necessary a problem but the converter has been much tested under *nix env

Does this generate csv files ? (note: the > is not necessary)

May we exchange by mail ? contact through the App page, that will be easier

nkrabbe_splunk
Splunk Employee
Splunk Employee

When I ran the cat | ./nmon2csv.pl command it generated the following output:

2014-07-24 09:55:47
NMON file cksum: 794362084
2014-07-24 09:55:47
cksum unknown, let's convert data.
Thu Jul 24 09:55:47 EDT 2014
Begin processing file = /Applications/Splunk/etc/apps/nmon/var/spool/nmon2csv.41179.nmon
0: Finished get_nmon_data
0: Finished CONFIG
: Finished LPAR
: Finished CPU_ALL
: Finished FILE
: Finished MEM
: Finished PAGE
: Finished MEMNEW
: Finished MEMUSE
: Finished PROC
: Finished PROCSOL
: Finished VM
0: Finished TOP

etc....
0 Karma

nkrabbe_splunk
Splunk Employee
Splunk Employee

Thanks so much for the fast reply. Right now I am just copying some nmon data over to a single instance and monitoring the path. We are not using the shell script to actually collect the data locally on forwarders.

The first two searches you suggested both returned 0 results. I looked in the nmon data file itself and found the following line with ZZZZ in it:
ZZZZ,T0003,00:11:12,21-JUL-2014

0 Karma

guilmxm
SplunkTrust
SplunkTrust

Additionally, how to you collect nmon data from your Vios ? through forwarders and TA-nmon or using your own process to collect nmon files and use the App to monitor a place where you put these files ?

0 Karma
.conf21 CFS Extended through 5/20!

Don't miss your chance
to share your Splunk
wisdom in-person or
virtually at .conf21!

Call for Speakers has
been extended through
Thursday, 5/20!