Nmon app not collecting data from forwarders

[pkaranam]

Hi Team,

I am using NMON app for collect the CPU,Memory, Disk logs from forwarders and to identify the resource utilisation and more.

After configuring the app i was able to get the all required data for a week, but all of a sudden indexer stopped collecting the data which being sent by forwarders. Below are some error messages getting in indexer server. Could you please help me with this issue?

03-11-2017 17:18:28.427 +0000 ERROR ArchiveContext - From archive='/data/splunk/var/log/nmon/var/nmon_repository/splunkindexer_170311_1711.nmon': couldn't run "/data/splunk/etc/slave-apps/PA-nmon/bin/nmon2csv.sh": No such file or directory
03-11-2017 17:18:28.494 +0000 ERROR ArchiveContext - From archive='/data/splunk/var/log/nmon/var/nmon_repository/splunkindexer_170311_1711.nmon': couldn't run "/data/splunk/etc/slave-apps/PA-nmon/bin/nmon2csv.sh": No such file or directory
03-11-2017 17:18:28.498 +0000 ERROR ArchiveContext - From archive='/data/splunk/var/log/nmon/var/nmon_repository/splunkindexer_170311_1711.nmon': couldn't run "/data/splunk/etc/slave-apps/PA-nmon/bin/nmon2csv.sh": No such file or directory
03-11-2017 17:19:28.492 +0000 ERROR ArchiveContext - From archive='/data/splunk/var/log/nmon/var/nmon_repository/splunkindexer_170311_1711.nmon': couldn't run "/data/splunk/etc/slave-apps/PA-nmon/bin/nmon2csv.sh": No such file or directory
03-11-2017 17:19:28.559 +0000 ERROR ArchiveContext - From archive='/data/splunk/var/log/nmon/var/nmon_repository/splunkindexer_170311_1711.nmon': couldn't run "/data/splunk/etc/slave-apps/PA-nmon/bin/nmon2csv.sh": No such file or directory
03-11-2017 17:19:28.567 +0000 ERROR ArchiveContext - From archive='/data/splunk/var/log/nmon/var/nmon_repository/splunkindexer_170311_1711.nmon': couldn't run "/data/splunk/etc/slave-apps/PA-nmon/bin/nmon2csv.sh": No such file or directory
03-11-2017 17:20:28.514 +0000 ERROR ArchiveContext - From archive='/data/splunk/var/log/nmon/var/nmon_repository/splunkindexer_170311_1711.nmon': couldn't run "/data/splunk/etc/slave-apps/PA-nmon/bin/nmon2csv.sh": No such file or directory
03-11-2017 17:20:28.582 +0000 ERROR ArchiveContext - From archive='/data/splunk/var/log/nmon/var/nmon_repository/splunkindexer_170311_1711.nmon': couldn't run "/data/splunk/etc/slave-apps/PA-nmon/bin/nmon2csv.sh": No such file or directory
03-11-2017 17:20:28.585 +0000 ERROR ArchiveContext - From archive='/data/splunk/var/log/nmon/var/nmon_repository/splunkindexer_170311_1711.nmon': couldn't run "/data/splunk/etc/slave-apps/PA-nmon/bin/nmon2csv.sh": No such file or directory
03-11-2017 17:21:28.551 +0000 ERROR ArchiveContext - From archive='/data/splunk/var/log/nmon/var/nmon_repository/splunkindexer_170311_1711.nmon': couldn't run "/data/splunk/etc/slave-apps/PA-nmon/bin/nmon2csv.sh": No such file or directory
03-11-2017 17:21:28.619 +0000 ERROR ArchiveContext - From archive='/data/splunk/var/log/nmon/var/nmon_repository/splunkindexer_170311_1711.nmon': couldn't run "/data/splunk/etc/slave-apps/PA-nmon/bin/nmon2csv.sh": No such file or directory
03-11-2017 17:21:28.623 +0000 ERROR ArchiveContext - From archive='/data/splunk/var/log/nmon/var/nmon_repository/splunkindexer_170311_1711.nmon': couldn't run "/data/splunk/etc/slave-apps/PA-nmon/bin/nmon2csv.sh": No such file or directory
03-11-2017 17:22:28.576 +0000 ERROR ArchiveContext - From archive='/data/splunk/var/log/nmon/var/nmon_repository/splunkindexer_170311_1711.nmon': couldn't run "/data/splunk/etc/slave-apps/PA-nmon/bin/nmon2csv.sh": No such file or directory
03-11-2017 17:22:28.644 +0000 ERROR ArchiveContext - From archive='/data/splunk/var/log/nmon/var/nmon_repository/splunkindexer_170311_1711.nmon': couldn't run "/data/splunk/etc/slave-apps/PA-nmon/bin/nmon2csv.sh": No such file or directory
03-11-2017 17:22:28.647 +0000 ERROR ArchiveContext - From archive='/data/splunk/var/log/nmon/var/nmon_repository/splunkindexer_170311_1711.nmon': couldn't run "/data/splunk/etc/slave-apps/PA-nmon/bin/nmon2csv.sh": No such file or directory

Thanks!
Pavan

[guilmxm]

Hi !

From the splunkd messages, it seems you run indexers in cluster, is that correct ?

If so, can you check that the PA-nmon package has been correctly deployed ? splunkd messages seem to say that some content of the PA-nmon package is missing.

If you are running indexer in cluster, the PA-nmon package should be deployed by the master node when you apply the cluster bundle. (basically extracting the PA-nmon tgz archive in /opt/splunk/etc/master-apps/ and applying the bundle)

For reference, the deployment doc: http://nmon-for-splunk.readthedocs.io/en/latest/installation_distributed.html

Regards,

Guilhem