Why is the Universal Forwarder not loading Splunk Add-on for Unix and Linux?

[jcrawford26]

I'm working on deploying the Splunk Add-On for Unix and Linux to the universal forwarders in my environment using a configuration management system. I packaged the add-on into an RPM for easier management, which simply decompresses the archive into $SPLUNK_HOME/etc/apps so that I now have /opt/splunkforwarder/etc/apps/Splunk_TA_nix with the application - directories appserver, bin, etc. I've created a local directory and copied default/inputs.conf into it with inputs and enabled a number of the inputs. However, the single-node Splunk server, which does receive a number of other inputs from this forwarder, is not getting any of the inputs configured in the app.

I've examined the output from splunkd, and during startup it lists that it is reading in the various configuration stanzas in /opt/splunkforwarder/system/local/inputs.conf, but it does not output anything about any of the stanzas configured in the Splunk Add-on for Unix and Linux. This makes me think that it's completely ignoring the add-on, but I can't figure out why. I've checked and the add-on folder is owned by root but is all readable by Splunk. Any ideas as to why it's not working?

[jkat54]

You have to install the Splunk TA nix on the indexers too. It has an indexes.conf that configures the os index. If you read the documentation it says to install the TA almost everywhere..., forwarders, search heads, and indexers.

[DUThibault]

"If you read the documentation it says to install the TA almost everywhere..., forwarders, search heads, and indexers." Not at all. http://docs.splunk.com/Documentation/UnixAddOn/5.2.4/User/Platformandhardwarerequirements , under "Distributed installation of this add-on" states unequivocally that Splunk_TA_nix is required ONLY on forwarders. This is contradicted by the earlier http://docs.splunk.com/Documentation/UnixAddOn/5.2.4/User/DeploytheSplunkAdd-onforUnixandLinuxinadis... recommended installation locations. As the doc is self-contradicting, is it any wonder users are having problems?

[jkat54]

I don't see it that way at all. I think on one page it's talking about compatibility and on the other it's the "recommended" setup which is also technically correct. You can install the TA on one forwarder and not use the os index. You'd just modify the appropriate inputs.conf to point to another index. That part is up to you. But the easiest play is to just install it everywhere they recommend it.

Sometimes that can be an issue in an environment however... for example if you use the preconfigured indexes.conf that comes with the TA, it uses the $SPLUNK_DB variable, but your environment might make use of volumes instead.

[DUThibault]

"The easiest play [way?] is to just install it everywhere they recommend it." Indeed, and that may be what the http://docs.splunk.com/Documentation/UnixAddOn/5.2.4/User/DeploytheSplunkAdd-onforUnixandLinuxinadis... page indicates, but this is undermined by e.g.
http://docs.splunk.com/Documentation/UnixApp/5.2.3/User/WhataSplunkAppforUnixandLinuxdeploymentlooks... which clearly shows a supposedly working deployment with no Splunk_TA_nix on the indexer/search head.

On the other hand, http://docs.splunk.com/Documentation/UnixApp/5.2.3/User/DeploytheSplunkAppforUnixandLinuxinadistribu... , under "Steps to building a Splunk App for Unix and Linux deployment", step 6 states the add-on is to be installed on the search heads.

The http://docs.splunk.com/Documentation/UnixAddOn/5.2.4/User/Platformandhardwarerequirements page is not about "compatibility", it's about requirements. If the add-on were not required on the search head, the app would work without it. It does not.

When Splunk_TA_nix is installed on a forwarder, it uses index=os, right? And the App, on its Settings page, clearly shows Unix Index(es) = "index=os" (by default). So it does not appear (to me , at least) that it's a matter of index naming. It appears to be some sort of data conversion or labelling black magic that the add-on does. Keep in mind that a newbie like me considers Splunk as a big black box: installation instructions and application advisories should be written appropriately. It was very frustrating to install the app (without Splunk_TA_nix on the search head) and have absolutely no clue as to what was broken. Splunk was not telling me what needed fixing.

[alemarzu]

Hi @jcrawfordd26

Try giving execution permissions to splunk user/group too, many of this inputs are scripts.

[jcrawford26]

I've just checked this, all of the scripts are world-executable and work just fine when I execute them. Even if the scripts were not functioning, I would think something would be logged when it read the configuration stanzas for them? I am fairly certain that the root of the problem is that Splunk is not reading the app's inputs.conf at all.

[esix_splunk]

You can confirm this by running btool on the forwarder in question

./splunk btool inputs list  --debug | grep TA_nix

That should show your inputs, if they are being used. I would also recommend checking the permissions on the nix TA directory on the client and make sure they are correct.

[DUThibault]

I'm having the same problem. I first had Splunk_TA_nix and splunk_app_for_nix deployed on my Splunk instance and its forwarders, and that worked fine. But I wanted to have the data inputs exclude the server (and if you disable the scripts on the server, the deployment service disables them on the forwarders too), so I now have a server with splunk_app_for_nix and the forwarders with Splunk_TA_nix. I've run Splunk_TA_nix/bin/setup.sh on the forwarders to enable just one source type to start with. The Splunk server receives some data but throws it away with this message:

Received event for unconfigured/disabled/deleted index=os
with source="source::bandwidth" host="host::dut-centos7"
sourcetype="sourcetype::bandwidth". So far received events
from 1 missing index(es).

Unlike before, the Splunk_TA_nix scripts don't show up in the Source types screen. splunk_app_for_nix has been run and configured, so why is the 'os' index not created?

[ddrillic]

You better open a new thread and refer to this one as reference...

[DUThibault]

See https://answers.splunk.com/answers/521110/how-to-install-the-splunk-add-on-for-unix-and-linu.html#an... for the answer.