Many of the Splunk Add-on for Unix and Linux data collection inputs are scripted inputs. The scripts smooth over variations to the input that arise across various Unix and Linux platforms. Scripted input means that Splunk runs the same commands you run on the terminal.
If a needed Unix command is not installed or lacks proper permissions, then Splunk can't collect the associated data. In other words, the Unix commands and access to them are prerequisite for collecting data. See the scripted inputs table in the Splunk Add-on for Unix and Linux manual for the underlying UNIX commands.
There are a few inputs in the Splunk Add-on for Unix and Linux that monitor files. You can find best practices for which UNIX permissions to use and how in the Splunk Answers topic Which UNIX permissions are best for monitoring files?
As you deploy the Splunk Add-on for Unix and Linux across many hosts, you may find it impractical to check each and every server for errors as the scripted inputs run. Luckily, Splunk excels at solving that very problem!
Apply the Best practice: Forward search head data to the indexer layer principle not just to the search heads, but to any add-on deployment destination. If you're worried about the amount of disk space this might consume, then follow this practice only during the process of deploying the Splunk Add-on for Unix and Linux.
Once those instances are forwarding their Splunk data, it should be trivial to find issues within the
_internal index. Try this search to identify problem hosts:
index=internal sourcetype=splunkd source=*splunkd.log loglevel=ERROR component=ExecProcessor SplunkTAnix
As you pinpoint issues, use the instructions outlined in Troubleshoot the Splunk Add-on for Unix and Linux to debug further.
If you're using a Windows system as a deployment server, you may experience end-of-line (eol) issues once the scripted inputs of SplunkTAnix attempt to execute on the UNIX clients. Each line in a text file on Windows ends with a carriage return (
\r) and then a new line (
\n), whereas on UNIX, lines end only with the new line (
Splunk is able to reconcile the differences when it reads the configuration files. But the UNIX shells themselves will throw all sorts of misleading errors rather than ignoring the carriage returns. A symptom of this might look like the thread described in the Splunk Answers post SplunkTAnix cannot open scripts.
Resolve this by converting the line endings to UNIX format at its location on the deployment server Windows host. See the Splunk Answers topic SplunkTAnix cannot open scripts for solutions on how to fix the files on the Windows machine before the deployment server distributes to the clients. Keep in mind that manual text edits of those files may reintroduce the problem.
Another good Splunk Answers topic for Windows considerations is What are the pain points with deploying your Splunk architecture on Windows OS?
Rather than listing out every UNIX host in the deployment server, embrace the wildcards (
*) in the
machineTypesFilter. For details, see set up client filters in the Updating Splunk Enterprise Instances manual.
As your distribution becomes more distributed, you may find it useful to review the Splunk Answers topic What are best practices for deploying an add-on with slightly different configurations?
If you're noticing a few options of how to collect the UNIX data, then take a gander at the Splunk Answers topic What's the best practice for collecting UNIX OS data? Tangentially related is the post What are the best practices to create a local configuration file?
Anything else to add? Anything you'd do differently?