NMON Splunk app - Monitor AIX files - local instance for testing purpose

[andrewpagans]

Ciao All,

We are trying to test Splunk NMON app but we are not able to index the data.
Our configuration is in local on Ubuntu Linux Virtual Machine (so a configuration without Splunk forwarders), manually picking up, from the AIX machine, the generated raw logs files from nmon tool.

By now Splunk just indexed the Ubuntu sample nmon data present by default into the Splunk NMON app.

We followed the guides here below:
http://nmonsplunk.wikidot.com/documentation:installation:standalone
http://youresuchageek.blogspot.it/2014/04/nmon-for-splunk-performance-monitor-for.html

So basically we have:
1. Installed Splunk NMON app
2. Created the index “nmon”
3. Unzip the archive of TA-nmon present in /opt/splunk/etc/apps/nmon/resources
4. Created the directory /opt/splunk/etc/apps/nmon/nmon-repository
5. Put our .nmon files here
6. Created the inputs.config into /opt/splunk/etc/apps/nmon/local with below information:

[monitor:///opt/splunk/etc/apps/nmon/nmon-repository/*nmon]
disabled = false
index = nmon
sourcetype = nmon_processing
crcSalt = <SOURCE>

Then we have restarted Splunk but still not working.

Checking further on the app folders we understood where the Ubuntu sample data where and so we had:
1. Moved our AIX .nmon file into /opt/splunk/var/log/nmon/var/nmon_repository
2. After some time the files has been picked up
3. A new folder with the hostname of the .nmon file has automatically created under /opt/splunk/var/log/nmon/var containing several files
4. but nothing appears in the Nmon Splunk app

May you please help us to understand where we are performing some wrong set-up or operation?

Thank a lot

[guilmxm · Accepted solution]

Hello !

Ok, let's restart from the beginning, both the links you have used are out of date, and should not be used.

The official documentation is:

http://nmon-for-splunk.readthedocs.io/

Now, the installation and the deployment are much more simple than that:

• Install the core application in your standalone server
• Ensure your Universal Forwarders are correctly configured (eg. they do have an outputs.conf and forward data to your indexers)
• Create the nmon index
• Untar the content of the TA-nmon archive to your servers (using the deployment server, or manually, or using your configuration management, up to you)
• Restart the UF

Nothing more to do, the data collection and indexing are activated by default in the TA-nmon.

Please follow the trouble shooting guide:

http://nmon-for-splunk.readthedocs.io/en/latest/Userguide.html#troubleshooting-guide-from-a-to-z

Note that you can as well test the deployment (Linux only) using Ansible and Vagrant, on the fly:

https://github.com/guilhemmarchand/splunk-vagrant-ansible-collections

Guilhem

[edoardo_vicendo]

Ciao Guilhem,

First of all thanks for your time and reply, I am working with Andrea (andrewpagans) to test the NMON Splunk app.
Based on your post, we are now configuring the forwarder (in our local VM) and so we have:

1. Downloaded and Installed the Universal Forwarder on the port 8090 (because 8089 already occupied by Splunk Enterprise)

2. Followed the guide at http://docs.splunk.com/Documentation/Forwarder/6.5.2/Forwarder/Configuretheuniversalforwarder, here below command executed via shell in /opt/splunkforwarder/bin:

   ./splunk add forward-server 127.0.0.1:9997
   ./splunk set deploy-poll 127.0.0.1:8089
   ./splunk add monitor /var/log
   ./splunk restart

We have also seen the video tutorial but we are still unable to view the forwarder options in Splunk Enterprise web interface.

Please consider we are new to Splunk installation and we are also a little bit confused as we were expecting to be in this scenario (http://nmon-for-splunk.readthedocs.io/en/latest/installation_standalone.html#installation-for-standa...) and we were not thinking to install forwarders, and even the folder /client-config in the path /opt/splunk/etc/deployment-apps as showed in the video tutorial is not present at the moment in our environment.

Are we going on the right way 🙂 ?

Thanks a lot,
Edoardo

[guilmxm]

Hi Eduardo,

Ok, I see.

To answer your question, the "client-config" shown in the video is a base application, basically it is just being used to configure the Universal Forwarder output configuration (a Universal Forwarder needs to know where to send its data, this is being stored in a file called outputs.conf, and the command ./splunk set deploy-poll creates this file for you)

You are not in this scenario because you intend to run a Universal Forwarder on the same host than the one running your Splunk instance, the standalone doc expects you to run a standalone Splunk instance and remote hosts running Universal Forwarders.

On a standalone instance, you basically just need to untar the content of the TA-nmon archive into /opt/splunk/etc/apps/ to get the performance collection working on the host.

However, this should work, the thing you want to get first is the link between your Universal Forwarder instance and your local Splunk instance for the deployment (What Splunk call Phone Home, UF --> Splunk on TCP 8089 by default for the deployment server)

Go to:

Settings / DISTRIBUTED ENVIRONMENT / Forwarder management

Can you see your Universal Forwarder instance ?

The main log you want to check is "splunkd.log", on the UF check:

/opt/splunkforwarder/var/log/splunk/splunkd.log

If for any reason there is a network failure (firewall...) you should see it from traces

[edoardo_vicendo]

Hi Guilhem,

Thanks to your indication we were able to understand why, checking on the /opt/splunkforwarder/var/log/splunk/splunkd.log, please find here below the error:

03-03-2017 16:12:55.635 +0100 WARN  TcpOutputFd - Connect to 127.0.0.1:9997 failed. Connection refused
03-03-2017 16:12:55.635 +0100 ERROR TcpOutputFd - Connection to host=127.0.0.1:9997 failed

The problem is due to the fact that the license expired and was automatically upgraded to a "Free License" and looking at the following link http://docs.splunk.com/Documentation/Splunk/6.5.2/Admin/TypesofSplunklicenses this kind of license do not foresee to have a forwarder. I'll check if feasible to replace with a Dev/Test License.

By now really thanks for your valuable support.

Best Regards,
Edoardo

[edoardo_vicendo]

Hi Guilhem,

I started over with a new VM and now I am one step forward, I am able to see the "Forwarder Management" page with 1 Client but no "Apps" and no "Server Classes".

I am going to read the Forwarder Manual because there is still something I am missing in the set-up.

http://docs.splunk.com/Documentation/Forwarder/6.5.2/Forwarder/HowtoforwarddatatoSplunkEnterprise

I'll keep you posted.

Best Regards,
Edoardo

[guilmxm]

Hello Edoardo,

• Uncompress the TA-nmon tgz archive in /opt/splunk/etc/deployment-apps/
• In the Forwarder management, you will now see the TA-nmon in the apps tab
• Create a serverclass that matches your hosts (you can use a * to match all hosts), and add the application, VERY IMPORTANT: Ensure to check the box "restart splunkd" in the app configuration.

That's the process to deploy apps using the Splunk server 😉