Splunk Add-on for Unix and Linux

[joewetzel63]

Hi All,

I've installed the Splunk Add-on for Unix and Linux in both Splunk Enterprise as well as my forwarder which is running 9.3.2  However I keep running into this error below:

12-19-2024 15:54:30.303 +0000 ERROR ExecProcessor [1376795 ExecProcessor] - message from "/opt/splunkforwarder/etc/apps/Splunk_TA_nix/bin/vmstat_metric.sh" /opt/splunkforwarder/etc/apps/Splunk_TA_nix/bin/hardware.sh: line 62: /opt/splunkforwarder/var/run/splunk/tmp/unix_hardware_error_tmpfile: No such file or directory

The above is coming from the splunkd.log after I have stopped and restarted the SplunkForwarder.service.  I am very new to Splunk and do not posses any certifications.  My company has tasked me with learning and configuring Splunk and I am enjoying it except I am unable to get this data sent to my indexer so that I can see the data in Search and Reporting.

These are the steps taken so far:

1. Installed Splunk Add-on for Unix and Linux on my enterprise UI machine
2. installed Splunk Add-on for Unix and Linux on my unvf
3. As the local directory was not created at "/opt/splunkforwarder/etc/apps/Splunk_TA_nix/" I created it and copied the inputs.conf from default to local and then allowed the scripts I wanted.
4. Made sure the Splunk user was owner and had the privileges needed to the local directory
5. stopped the splunk service and restarted it.
6. Ran -  cat /opt/splunkforwarder/var/log/splunk/splunkd.log | grep ERROR
7. Almost every Error is "unix_hardware_error_tmpfile: No such file or directory"
8. if I create the tmpfile it disappears and is not recreated

I'm sure there are many other things I didnt mention because I honestly dont remember because I have been trying to figure this issue out since yesterday and am not getting anywhere. 

PLEASE HELP!

[scelikok]

Hi @joewetzel63,

In the error message it complains about "/opt/splunkforwarder/var/run/splunk/tmp/unix_hardware_error_tmpfile" file. This tmp folder does not exist on default, that is why it cannot create unix_hardware_error_tmpfile file. You can try creating /opt/splunkforwarder/var/run/splunk/tmp folder.

When I checked the addon (v9.2.0) it uses correct path as "$SPLUNK_HOME/var/run/splunk/unix_hardware_error_tmpfile". 

Can you confirm and try using the latest version of the addn? 

[jw220635]

After recently upgrading the Splunk_TA_nix to version 9.2.0, I'm seeing the same issue.  Has anyone fixed this issue?

[christinetran]

I think it's because hardware.sh is called by both vmware.sh and vmware_metric.sh, and hardware.sh has an rm $TMP_ERROR_FILTER_FILE statement just below the grep statement (where the error is generated.)

If you are running both vmware.sh and vmware_metric.sh at the same periodicity (for example 60 seconds), it's a race condition when both scripts invoke hardware.sh which has an "rm", which would remove the file before the second script get to that line of execution (the grep).  If you look at your error log, the error is logged against vmware.sh or vmware_metric.sh, but not both.  I ran into this issue, and I added a few seconds offset, for example, run vmware.sh at 60 seconds interval and vmware_metric.sh at 65 seconds interval, to eliminate the race condition.

[PickleRick]

This calls for a bug report, don't you think? It's a very bad practice if a script relies on a statically named file.