I have a POC Splunk setup running the free license. Getting data from several linux hosts running a universal forwarder with no issues. I went and installed the Splunk for *NIX app via the console and restarted the server. Went into the Splunk Addon for *NIX config and enabled a few metrics like CPU and memory scripts. Only seeing data for the Splunk server itself, not the universal forwarder. Trying to follow the documentation and it appears to say it needs to be installed on the forwarder as well so I expanded the tar file for the app into the etc/apps folder on the universal forwarder and restarted it. Still not seeing any data from the forwarder. Is there any other config I need to perform or am I missing something?
Is the documentation topic you are looking at Install the Splunk App for Unix and Linux in a distributed Splunk environment? You need to install the add-on on the various tiers in your deployment. You might also need the supporting add-on, if you are forwarding data from your search head to your indexer. You also have to configure the indexer to receive the data and the forwarders to send it.
Originally I had installed the Splunk for *NIX app instead of the addon on the forwarder. I shut down the forwarder, deleted the app, expanded the addon to where I now have the directory $splunk_home/etc/apps/Splunk_TA_nix on my forwarder. I started the forwarder again, but am still seeing no data coming in from the forwarder. My monitored log directories for tomcat apps are sending data fine. Just not seeing the data from the *nix app. I've enabled cpu, iostat and vmstat in the addon config in the console. Not seeing any errors jumping out at me in the forwarder logs either. When I search for Splunk_TA_nix in the forwarder logs I get nothing either. Maybe the addon isn't loading for some reason?
Usually when you don't see data, it's network, permissions, or a combination of both.
Any logs would be beneficial, particularly on the forwarders.
There is no network connectivity issues from my perspective. Splunk forwarder is successfully sending application (tomcat) logs to the indexer on port 9997. I followed the instructions using the command line method here. I've uploaded the fowarder's splunkd log here. Below are the contents of the forwarder's app directory.
-bash-3.2$ ls -l /apps/splunkforwarder/etc/apps/
total 20
drwxr-xr-x 4 user user 4096 Oct 22 20:05 introspection_generator_addon
drwxr-xr-x 5 user user 4096 Nov 7 16:29 learned
drwxr-xr-x 5 user user 4096 Oct 22 20:05 search
drwxrwxr-x 9 user user 4096 Nov 10 14:34 Splunk_TA_nix
drwxr-xr-x 4 user user 4096 Oct 22 20:05 SplunkUniversalForwarder
What exactly are you searching for on the search head?
Can we see the splunkd.log on the indexer?
Searches like below are working fine:
sourcetype=debs
When I search the os index that the Splunk for *Nix app sends data to I'm only seeing data from the indexer host. The below example search only shows data for the indexer.
index=os
I've uploaded the indexer logs below.
indexer log part1
indexer log part2