Hello, I have single server which has Splunk Enterprise installed. My requirement is to monitor some linux hosts in our network, have them send performance data like CPU/Memory/DISK stats etc to the Splunk server. I have installed the Splunk App and Add-on for Unix and Linux (*NIX App and *NIX Add-on) my Splunk server. I also went ahead and installed the Universal Forwarder on one of my Linux hosts. What's next ? I am not getting any CPU/MEMORY/DISK data in my Splunk dashboard from the linux host. When i click on the "Splunk App for Unix and Linux" app in my Splunk dashboard it shows empty .
I have looked at the official documentation for the Splunk Add-on for Unix and Linux. It talks about installing the Add-on on the Universal Forwarder. This did not make sense to me, as i have a dozen Linux hosts in my environment , so are we supposed to install add-on on each n every host we want to monitor? Also, the installation instructions for" add-on" say on one hand to install it on the universal forwarder and on the other hand it says post install login to the Splunk interface and enable/disable the parameters, scripted inputs etc you want to monitor. The Universal Forwarder does not even have a Splunk Web interface .
If the add-on does need to be installed on each and every device we want to monitor, how do we configure the options like what inputs to monitor when there is no web interface on the device (which also has the universal forwarder installed)?
As an alternate the document talks about running the setup of add-on via command line. So i went ahead and ran the below as shown in their documentation on my linux host :--
$SPLUNKHOME/bin/splunk cmd $SPLUNKHOME/etc/apps/SplunkTAnix/bin/setup.sh*
This command then asks for splunk username and password.
If i enter her my splunk server interface admin credentials it says LOGIN Failed. Also, I have not setup any credentials when i installed the univ. forwarder on this host, so if i leave the username/pwd empty, it says LOGIN Failed again. What creds is it really expecting ?
Any help will be appreciated.
Yes, that add-on needs to be installed on UFs. You can deploy that using deployment server.
hope this helps.
Nilesh . Thanks for the response. Does the Add-on also need to be deployed on the central Splunk server ?
I have deployed the add-on in both places ( UF's and Central Splunk server). Will the input options not conflict with one another ? For instance, lets say if i enable scripted input of "cpu.sh" on Central but not on the UF side, what will be the end result ? Will i get the cpu metrics of the UF server ?
I am still not getting any performance metrics from my Universal forwarder. I have followed the steps as per the documetation. Not sure what's missing.
Add-on needs to be installed on UF only. Install Splunk app for unix/linux on central Splunk server. restart Splunk on UF and see if any errors in splunked.log. also check inputs.conf under local folder, it should have stanzas as follows:
disabled = false
index = os
Nilesh, i do not have any inputs.conf file under local folder. I am referring to /opt/splunkforwarder/etc/apps/SplunkTAnix/local folder on my UF server. I had enabled all scripted inputs via command line method. For instance, after installing the Add-on on my UF, i ran the below commands as per the official documentation. This did not create any inputs.conf file under /local.
$SPLUNKHOME/bin/splunk cmd $SPLUNKHOME/etc/apps/SplunkTAnix/bin/setup.sh
This prompted me for admin credentials and showed a menu. I selected the " enable-all" option which i assumed enables all the options.
I did not use the scripted file method.
Neeraj - can you post cpu.sh stanza from inputs.conf ? also do you see any messages related with this add-on/app in Splunkd.log?
I downvoted this post because this "solution" slaves the server and forwarders, which does not match the distribution recommended in the documentation (which states splunktanix should be installed only on the forwarders). in particular, one could not then enable or disable a script data input on the server without having the setting immediately propagated to the forwarders.
I'm having the same problem. I first had SplunkTAnix and splunkappfornix deployed on my Splunk instance and its forwarders, and that worked fine. But I wanted to have the data inputs exclude the server (and if you disable the scripts on the server, the deployment service disables them on the forwarders too), so I now have a server with splunkappfornix and the forwarders with SplunkTAnix. I've run SplunkTAnix/bin/setup.sh on the forwarders to enable just one source type (bandwidth) to start with. The Splunk server receives some data but throws it away with this message:
Received event for unconfigured/disabled/deleted index=os with source="source::bandwidth" host="host::dut-centos7" sourcetype="sourcetype::bandwidth". So far received events from 1 missing index(es).
Unlike before, the SplunkTAnix scripts don't show up in the Source types screen. splunkappfor_nix has been run and configured, so why is the 'os' index not created?
Turns out http://docs.splunk.com/Documentation/UnixAddOn/5.2.4/User/Platformandhardwarerequirements is lying. If you install SplunkTAnix on forwarders, you must also install it on the server(s) that have installed splunkappfor_nix.
If, and only if, your server (the Splunk instance where splunkappfornix is installed) is to be monitored in exactly the same way as the clients (the machines where a Universal Forwarder and SplunkTAnix are installed), then you can install splunkappfornix and SplunkTAnix on the server and then deploy that to the clients. Be aware that despite having data inputs (scripts and monitored files and directories) listed separately in the Settings: (Data) Data inputs screen, you will NOT be able to change any setting (such as enabling/disabling) on one without having it change on the other.
If you want the server to merely run splunkappfornix and receive SplunkTAnix data from forwarders, you must install splunkappfornix AND SplunkTAnix on the server (leave SplunkTAnix's scripts and monitors disabled, their default setting), and then install SplunkTAnix on the forwarders (the clients). You will need to connect to each client and locally run SplunkTAnix/bin/setup.sh to enable the scripts and monitors. One key difference with this setup is that the forwarder data inputs will NOT show on the Settings: (Data) Data inputs screen.
Note that if you don't intend to change the settings often if at all, you can edit splunk-add-on-for-unix-and-linux524.tgz to change the "disabled=1" lines in SplunkTA_nix/default/inputs.conf before deploying the .tgz on the clients. This will save you from having to run setup.h.