All Apps and Add-ons

Does the Splunk App for Unix and Linux work with universal forwarders?

DFresh4130
Path Finder

I have a POC Splunk setup running the free license. Getting data from several linux hosts running a universal forwarder with no issues. I went and installed the Splunk for *NIX app via the console and restarted the server. Went into the Splunk Addon for *NIX config and enabled a few metrics like CPU and memory scripts. Only seeing data for the Splunk server itself, not the universal forwarder. Trying to follow the documentation and it appears to say it needs to be installed on the forwarder as well so I expanded the tar file for the app into the etc/apps folder on the universal forwarder and restarted it. Still not seeing any data from the forwarder. Is there any other config I need to perform or am I missing something?

0 Karma

ChrisG
Splunk Employee
Splunk Employee

Is the documentation topic you are looking at Install the Splunk App for Unix and Linux in a distributed Splunk environment? You need to install the add-on on the various tiers in your deployment. You might also need the supporting add-on, if you are forwarding data from your search head to your indexer. You also have to configure the indexer to receive the data and the forwarders to send it.

0 Karma

DFresh4130
Path Finder

Originally I had installed the Splunk for *NIX app instead of the addon on the forwarder. I shut down the forwarder, deleted the app, expanded the addon to where I now have the directory $splunk_home/etc/apps/Splunk_TA_nix on my forwarder. I started the forwarder again, but am still seeing no data coming in from the forwarder. My monitored log directories for tomcat apps are sending data fine. Just not seeing the data from the *nix app. I've enabled cpu, iostat and vmstat in the addon config in the console. Not seeing any errors jumping out at me in the forwarder logs either. When I search for Splunk_TA_nix in the forwarder logs I get nothing either. Maybe the addon isn't loading for some reason?

0 Karma

malmoore
Splunk Employee
Splunk Employee
  • Forwarder set up to send data to the receiving indexer on the correct port?
  • Indexer set up to receive data from forwarders on the same port?
  • Is connectivity good between all server(s)
  • Have firewalls either been turned off or configured to allow port(s) specified above?
  • TA installed in Splunk_TA_Nix on universal forwarder?
  • SA-Nix installed on server (search head) that runs the app (and on indexers when forwarding data from search head)?

Usually when you don't see data, it's network, permissions, or a combination of both.

Any logs would be beneficial, particularly on the forwarders.

DFresh4130
Path Finder

There is no network connectivity issues from my perspective. Splunk forwarder is successfully sending application (tomcat) logs to the indexer on port 9997. I followed the instructions using the command line method here. I've uploaded the fowarder's splunkd log here. Below are the contents of the forwarder's app directory.

-bash-3.2$ ls -l /apps/splunkforwarder/etc/apps/
total 20
drwxr-xr-x 4 user user 4096 Oct 22 20:05 introspection_generator_addon
drwxr-xr-x 5 user user 4096 Nov  7 16:29 learned
drwxr-xr-x 5 user user 4096 Oct 22 20:05 search
drwxrwxr-x 9 user user 4096 Nov 10 14:34 Splunk_TA_nix
drwxr-xr-x 4 user user 4096 Oct 22 20:05 SplunkUniversalForwarder
0 Karma

malmoore
Splunk Employee
Splunk Employee

What exactly are you searching for on the search head?
Can we see the splunkd.log on the indexer?

0 Karma

DFresh4130
Path Finder

Searches like below are working fine:

sourcetype=debs

When I search the os index that the Splunk for *Nix app sends data to I'm only seeing data from the indexer host. The below example search only shows data for the indexer.

index=os

I've uploaded the indexer logs below.
indexer log part1
indexer log part2

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...