All Apps and Add-ons

NMON for Splunk App - Looking for help and ideas to optimize CPU load on Forwarders, deport data conversion step to indexers

SplunkTrust
SplunkTrust

Hi,

I developed the App NMON for Splunk, performance monitor for AIX, Solaris and Linux Systems:

https://apps.splunk.com/app/1753/

Today, i am looking for help and idea to optimize and reduce to the minimum possible the CPU load on Universal Forwarders. (running TA-nmon, the forwarder version of the App)

To be short, one of ideas would be to deport the "Hard work" (which consists in data conversion through a third party script) from the Universal Forwarder to indexer(s))

Currently the App is working that way:

  • A third party script "nmon_helper.sh" launches the binary "nmon" periodically
  • This binary generates *.nmon files located in $APP/var/nmon_repository
  • Once a new *.nmon file is found by Splunk in this repository, then the Archive Processor calls a third party script "nmon2csv.pl" to convert the nmon file data into csv files in $APP/var/nmon_repository
  • Splunk permanently watches for new files within this directory, and index any csv files in batch mode (index and delete)

The reason why the App uses a third party script "nmon2csv.pl" to convert original nmon data into csv structured data resides in the very specific format of nmon data, even if this is a main goal for some future versions, this won't yet natively managed by Splunk.

That's why we use this conversion step.

But this configuration has the side effect of potentially generate CPU utilization on Forwarders hosts, because of this conversion step.

One of potential ideas to solve this would be to fully deport the "hard work" on indexers, instead of having this done on forwarders.

Currently, in Forwarders the configuration is set as following:

inputs.conf

##################################
#           nmon2csv stanza         #
##################################

# Source stanza for nmon2csv.pl script
# Associated with the source stanza within props.conf
# Every nmon file present within the directory will be converted into csv files
# Splunk can manage

[monitor:///opt/splunkforwarder/etc/apps/TA-nmon/var/nmon_repository/*nmon]

disabled = false
index = nmon
sourcetype = nmon_processing
crcSalt = <SOURCE>

####################################################
#           nmon csv converted files indexing           #
####################################################

# Every file present within this directory will be indexed then deleted
# This section should not be modified under normal use

[batch:///opt/splunkforwarder/etc/apps/TA-nmon/var/csv_repository/*nmon*.csv]

disabled = false
move_policy = sinkhole
recursive = false
crcSalt = <SOURCE>
index = nmon
sourcetype = nmon_data
source = nmon_data

####################################################
#                   nmon data collect                           #
####################################################

# These input script sanza will run nmon and generates nmon file 
# to be exploited by Splunk

# For AIX / Linux / Solaris

[script://./bin/nmon_helper.sh]
disabled = false
index = nmon
interval = 60
source = nmon_collect
sourcetype = nmon_collect

props.conf

##################################
#           nmon2csv stanza         #
##################################

# Source stanza for nmon2csv.pl script
# This source stanza will be called by the archive processor to convert NMON raw data into csv files
# SPlunk can manage. See inputs.conf for the associated monitor

[source::/opt/splunkforwarder/etc/apps/TA-nmon/var/nmon_repository/*nmon]

invalid_cause = archive
unarchive_cmd = $SPLUNK_HOME/etc/apps/TA-nmon/bin/nmon2csv.pl
sourcetype = nmon_processing
NO_BINARY_CHECK = true

The way it works and the way the application has been designed, implies that a nmon file can be managed on indexers no matters if it has been generated by a Splunk Forwarder running the TA-nmon (the lightweight App for forwarders) or by any other way (eg. manual run of nmon binary, etc...)

To illustrate this, you can simply copy any *.nmon files in $APP/var/nmon_repository of an indexer, Splunk will automatically convert it and index associated data.
And the data of the host that generated it will be available in views, and the associated host identified.

So, the idea would be to set the nmon forwarder to generate the nmon file and then to stream it to the indexer, and the indexer does the conversion step and indexes the generated data (when today the forwarder generates nmon files, converts them into csv and streams the resulting indexing data to indexers)

If i could use the forwarder to stream the content of nmon files in a way it would hold by the indexer, and the indexer would calls the nmon2csv.pl conversion script, this could solve any high CPU load in forwarders running the TA-nmon App.

Thanks in advance for any help or welcomed ideas !

Guilhem

0 Karma
1 Solution

SplunkTrust
SplunkTrust

Out of date question, the Nmon App is quite stable now and indexing steps optimized as mush as possible

View solution in original post

0 Karma

SplunkTrust
SplunkTrust

Out of date question, the Nmon App is quite stable now and indexing steps optimized as mush as possible

View solution in original post

0 Karma