We are trying to test Splunk NMON app but we are not able to index the data.
Our configuration is in local on Ubuntu Linux Virtual Machine (so a configuration without Splunk forwarders), manually picking up, from the AIX machine, the generated raw logs files from nmon tool.
By now Splunk just indexed the Ubuntu sample nmon data present by default into the Splunk NMON app.
So basically we have:
1. Installed Splunk NMON app
2. Created the index “nmon”
3. Unzip the archive of TA-nmon present in /opt/splunk/etc/apps/nmon/resources
4. Created the directory /opt/splunk/etc/apps/nmon/nmon-repository
5. Put our .nmon files here
6. Created the inputs.config into /opt/splunk/etc/apps/nmon/local with below information:
[monitor:///opt/splunk/etc/apps/nmon/nmon-repository/*nmon] disabled = false index = nmon sourcetype = nmon_processing crcSalt = <SOURCE>
Then we have restarted Splunk but still not working.
Checking further on the app folders we understood where the Ubuntu sample data where and so we had:
1. Moved our AIX .nmon file into /opt/splunk/var/log/nmon/var/nmon_repository
2. After some time the files has been picked up
3. A new folder with the hostname of the .nmon file has automatically created under /opt/splunk/var/log/nmon/var containing several files
4. but nothing appears in the Nmon Splunk app
May you please help us to understand where we are performing some wrong set-up or operation?
Thank a lot
Ok, let's restart from the beginning, both the links you have used are out of date, and should not be used.
The official documentation is:
Now, the installation and the deployment are much more simple than that:
Nothing more to do, the data collection and indexing are activated by default in the TA-nmon.
Please follow the trouble shooting guide:
Note that you can as well test the deployment (Linux only) using Ansible and Vagrant, on the fly:
First of all thanks for your time and reply, I am working with Andrea (andrewpagans) to test the NMON Splunk app.
Based on your post, we are now configuring the forwarder (in our local VM) and so we have:
Followed the guide at http://docs.splunk.com/Documentation/Forwarder/6.5.2/Forwarder/Configuretheuniversalforwarder, here below command executed via shell in /opt/splunkforwarder/bin:
./splunk add forward-server 127.0.0.1:9997
./splunk set deploy-poll 127.0.0.1:8089
./splunk add monitor /var/log
We have also seen the video tutorial but we are still unable to view the forwarder options in Splunk Enterprise web interface.
Please consider we are new to Splunk installation and we are also a little bit confused as we were expecting to be in this scenario (http://nmon-for-splunk.readthedocs.io/en/latest/installation_standalone.html#installation-for-standa...) and we were not thinking to install forwarders, and even the folder /client-config in the path /opt/splunk/etc/deployment-apps as showed in the video tutorial is not present at the moment in our environment.
Are we going on the right way 🙂 ?
Thanks a lot,
Ok, I see.
To answer your question, the "client-config" shown in the video is a base application, basically it is just being used to configure the Universal Forwarder output configuration (a Universal Forwarder needs to know where to send its data, this is being stored in a file called outputs.conf, and the command ./splunk set deploy-poll creates this file for you)
You are not in this scenario because you intend to run a Universal Forwarder on the same host than the one running your Splunk instance, the standalone doc expects you to run a standalone Splunk instance and remote hosts running Universal Forwarders.
On a standalone instance, you basically just need to untar the content of the TA-nmon archive into /opt/splunk/etc/apps/ to get the performance collection working on the host.
However, this should work, the thing you want to get first is the link between your Universal Forwarder instance and your local Splunk instance for the deployment (What Splunk call Phone Home, UF --> Splunk on TCP 8089 by default for the deployment server)
Settings / DISTRIBUTED ENVIRONMENT / Forwarder management
Can you see your Universal Forwarder instance ?
The main log you want to check is "splunkd.log", on the UF check:
If for any reason there is a network failure (firewall...) you should see it from traces
Thanks to your indication we were able to understand why, checking on the /opt/splunkforwarder/var/log/splunk/splunkd.log, please find here below the error:
03-03-2017 16:12:55.635 +0100 WARN TcpOutputFd - Connect to 127.0.0.1:9997 failed. Connection refused 03-03-2017 16:12:55.635 +0100 ERROR TcpOutputFd - Connection to host=127.0.0.1:9997 failed
The problem is due to the fact that the license expired and was automatically upgraded to a "Free License" and looking at the following link http://docs.splunk.com/Documentation/Splunk/6.5.2/Admin/TypesofSplunklicenses this kind of license do not foresee to have a forwarder. I'll check if feasible to replace with a Dev/Test License.
By now really thanks for your valuable support.
I started over with a new VM and now I am one step forward, I am able to see the "Forwarder Management" page with 1 Client but no "Apps" and no "Server Classes".
I am going to read the Forwarder Manual because there is still something I am missing in the set-up.
I'll keep you posted.
That's the process to deploy apps using the Splunk server 😉