NMON Performance Monitor for Unix and Linux Systems: Is there a way to customize an NMON index on a per team basis?

[dmacgillivray]

Hello Guilhem & NMON community,

We would like to somehow modify the NMON app to post NMON data to different indexes that associate with the dev or qa team that would use it.

IE:
(INDEXER) __index=nmon(team1) ---> (SEARCHHEAD (team1)nmon_app1
(INDEXER) __index=nmon(team2) ---> (SERRCHHEAD (team2)nmon_app2

Does the above make sense? Basically, we just want to be able to put a dev or qa team's data in the same index that they use for other sourcetypes, as we have almost too many teams using Splunk to be combined into one primary index=nmon.

Experiences or suggestions welcome.

Thank You,
Daniel MacGillivray

[guilmxm · Accepted solution]

Hi Daniel,

That's an interesting question and case !
I think you can follow the advice above, with some supplementary information:

1. UF package customization

A Python customization tool is provided in the resource directory:

http://nmonsplunk.wikidot.com/documentation:userguide:configure:packaging:createagent

You could use that tool to create automatically 2 or more UF package that will use different indexes to stream data to your indexers, and you will manage the deployment depending on your wish (let's say an UF-nmon-qa and UF-nmon-dev)

The tool allows you to change the destination index. (see usage in the Wiki page)

2. App customization

An other Python tool is provided to manage global Application customization, this Python script will change every required file to match you custom index:

http://nmonsplunk.wikidot.com/documentation:userguide:configure:packaging:customize

The idea would be to specify an "nmon_*" as your custom index (i haven't tested this but i think it can work the current script version, in the worst case an update of the script would make that work)

This would create a custom spl package containing the Application, using "nmon_*" as the index name will modify every search, view, datamodel and will match any index starting by nmon_

The only thing you will have to manage yourself will be to ensure you have correct configuration for your indexers (an indexes.conf containing both indexes), the current customization tool will not manage this but you can easily correct the generated PA package.

Note that using these tools is fully supported (by me 🙂 and should be used to follow application future updates, such you just have to redo these steps to create your own packages ready for deployment.

Guilhem

EDIT:

This customization scenario has been qualified and documented under the Nmon wiki:

http://nmonsplunk.wikidot.com/documentation:userguide:configure:splitindex

This procedure will allow managing different scenarios using Python supported customization tools: Splitting default index into multiple indexes to manage various configuration such as other environments (each index can have its own retention or parameters), Managing Nmon data from the central Core application targeting various indexers configuration (all data indexed in same indexer or indexer cluster or each environment with its own indexer(s) and so on)

[dmacgillivray]

Hi Guilhem, This is an awesome solution as the global solution will work perfectly. I just realized and correct me if I am wrong,. the only other thing that may have to be manually adjusted is the .png NMON icon for the app itself?

Thanks Again,
Daniel

[dmacgillivray]

Hi Guilhem . The scripts for app customization and UF package customization are doing what they were intended to do. There are two exceptions ...... I was able to get around the first one.

1) The app customization relies on 2.7 python. I set the .py script to read #/opt/splunk/bin/python

2) The icons for CPU, Process, Disk , Application Information and Safe Center do not show up after customization.
I am sure there is a way around it but the indexes and searches are running ok though.

Daniel

[dmacgillivray]

I believe the issue with number two possibly was the command switch combination I used to create the customized app. Not sure though. Just wanted to be more clear on what I did..

1) Ran the command

./Nmon_SplunkApp_Customize.py -f nmon-performance-monitor-for-unix-and-linux-systems_1603.tgz -i qa -r nmon_qa

2) Resulting output affected all the xml pages. Seem's ok otherwise...
IE:

For now sed -i 's/nmon_qa_qa/nmon_qa/g' . will fix the issue, but I would not recommend this. It for me was a workaround. I would guess the command switch would need some adjustment or I am doing it wrong 🙂

[dmacgillivray]

Here is the output that was bad.. Sorry for any confusion.

[guilmxm]

Hi Daniel,

To answer:

1. Python

They are Python script indeed, and they do require Python 2.7.x (embedded with any quite recent Linux OS)
You call them like any script (./myscript.py or python myscript.py)

If you don't have Python 2.7 but run them in the same server than a splunk installation, you could call it that way:

/opt/splunk/bin/splunk cmd python /myscript.py

Please give a valid argument to each switch, this is the root cause of your issue

./Nmon_SplunkApp_Customize.py --help
usage: Nmon_SplunkApp_Customize.py [-h] [-f INFILE] [-i INDEX_NAME]
[-r ROOT_DIR] [-a TA_NMON] [-p PA_NMON]
[--csvrepo CSV_REPOSITORY]
[--configrepo CONFIG_REPOSITORY]
[--version]

optional arguments:
-h, --help show this help message and exit
-f INFILE Name of the Nmon Splunk APP tgz Archive file
-i INDEX_NAME Customize the Application Index Name (default: nmon)
-r ROOT_DIR Customize the Application Root Directory (default:
nmon)
-a TA_NMON Customize the TA NMON Root Directory (default: TA-
nmon)
-p PA_NMON Customize the PA NMON Root Directory (default: PA-
nmon)
--csvrepo CSV_REPOSITORY
Customize the local CSV Repository (default:
csv_repository)
--configrepo CONFIG_REPOSITORY
Customize the local Config Repository (default:
config_repository)
--version show program's version number and exit

So if you want to:

• change the index name: -i
• change the App root directory: -r

[guilmxm]

Please double check:

http://nmonsplunk.wikidot.com/documentation:userguide:configure:splitindex

http://nmonsplunk.wikidot.com/documentation:userguide:configure:packaging:customize

We can exchange by mail for easier help (comments are hard to read and write):
http://nmonsplunk.wikidot.com/about-support

I am wondering about what you are trying to get, is in your intention to have custom version of the App for you qa environment in the same place than the standard App ?

I don't recommend this, this is wasting resources and you can easily manage various indexes for your different env like i exposed in the scenario article

[guilmxm]

Also consider keeping up to date to last release of the App, currently 1.6.06 🙂

[guilmxm]

Hello Daniel,

Have you been able to deal with your custom scenario ?

If not, don't hesitate to contact me by mail (guilhem.marchand@gmail.com) and we will manage this together.

Guilhem

[guilmxm]

Daniel,

No, you should not have anything to manage using customization tools, they will do that job for you.

Just to sum up the scenario:

• Indexer(s) standalone or in cluster will contain at least 2 nmon indexes, let's say nmon_qa and nmon_dev (thanks tskinnerrivsec for you smart suggestion)
• search head (standalone or sh cluster) will contain a customized Nmon App
• TA-nmon package will be created to be deployed on different clients, qualification and dev for example

If you let me very few time, i will qualify this scenario and write an customization scenario in the Wiki, and will let you know.

[dmacgillivray]

Thanks everyone for your help today.

[guilmxm]

Daniel,

I have edited my answer below with a link to the small doc i have just created (its a first version)
My tests were ok, the application will perfectly manage different indexes using Python customization tools. (using a prefix common to all indexes, such as "nmon_" for example)
Then, you need to create your own TA-nmon to address each of your case and you will be good.

Don't hesitate to revert 🙂

[tskinnerivsec]

you could always install a 2nd copy of the app on the same search head. You would just want to untar the 2nd copy to a different directory than /opt/splunk/etc/apps/nmon (which is the default install location) Also, you'd probably want to customize the apps.conf file:

[package]
id = nmon
check_for_updates = true

[ui]
is_visible = 1
label = NMON Performance

You would probably want to customize the id, and label strings here to differentiate the versions of the app.

Data models shouldn't need to be customized, you will just adjust data model searches by including filter | search index=nmon...

This is probably a bit of an inconvenience, but certainly one method of addressing your problem. You could also stand up an additional search head for one of the teams and that would cut down on the number of customizations you need to make.

[guilmxm]

@tskinnerivsec

I would not recommend that, this would be over complicated and will waste system resources on indexers and search heads.
As a complete solution, the Nmon app comes with many views, variety of configuration and data models, scheduled savedsearches (to generate inventory data, alerting...) and so on.
Duplicating the application would cost a lot to Splunk infrastructure, customizing the App is i think a better approach.

Guilhem

[tskinnerivsec]

Do your dev and qa teams work on different servers? If so, you could configure inputs.conf file in UF on qa server to send data to index=nmon_qa and dev server to send data to index=nmon_dev. Then you would just have to make sure to adjust your searches to include a filter for your specific index.

[dmacgillivray]

Excellent that make complete sense. Will try that out and thanks !!

[dmacgillivray]

Thanks "tskinnerrivsec" for checking back. They do utilize different servers, so your suggestion makes sense. I had actually done this with the old TA-nix app prior to NMON and it worked out for the best.

With NMON I had some reservations on changing the front end of the app, as I would guess data models, then searches and -- then I am not sure if two or three differently named versions of the app can be on the same Search Head?

In reality, just having two would be enough of a workaround as one group of servers is classified as sensitive but not sure it can be done...

Thanks,
Daniel