Configured .sh scripts not being run/reporting data

[joewetzel63]

I created .sh scripts that do the following:

#!/bin/bash

# Name of the service to monitor
SERVICE_NAME="tomcat9"

# Check if the service is running
SERVICE_STATUS=$(systemctl is-active "$SERVICE_NAME.service")

# Output status for Splunk
if [ "$SERVICE_STATUS" == "active" ]; then
echo "$(date): Service $SERVICE_NAME is running."
else
echo "$(date): Service $SERVICE_NAME is NOT running."
fi

The above is obviously what Im using for Tomcat but I have others all doing the thing just different service names. These scripts reside in:

/opt/splunkforwarder/bin/scripts

Additionally I have configured these scripts to be run in /opt/splunkforwarder/etc/system/local/inputs.conf an example of what that looks like is below:

[script:///opt/splunkforwarder/bin/scripts/monitor_service_<service_name>.sh]

disabled = false
interval = 60
index = services
sourcetype = service_status

As you can see I also have configured the following:

index = services
sourcetype = service_status

These are also configured in Splunk Enterprise respectively and the index is configured for Search, in linux  Splunk is the owner and the group is also Splunk. Additionally all of the scripts are executable and successfully run when I test them, however none of this data seems to be passed from the forwarder as none of the expected data is returned including the recognition of the index and sourcetype in Search.  Additionally I have attached a screen capture of splunkd.log showing the scripts as being recognized.  

[splunkreal]

Hello, as best practise you should create and deploy an app from deployment server with your inputs.conf and script. Also make sure you include a valid timestamp at the beginning of the output in US format.

Follow these instructions : https://dev.splunk.com/enterprise/docs/developapps/manageknowledge/custominputs/scriptedinputsexampl...

[PickleRick]

Yes, packaging your content into an app is a good practice but it shouldn't matter much if it's in apps/<app>/local or system/local for actually running the config (unless the settings get overwritten of course).

And no, timestamp doesn't have to be in US format. That's what time parsing sourcetype settings are for.

But back to the @joewetzel63 's issue - did you try running the script "as Splunk"? With

splunk cmd /opt/splunkforwarder/bin/scripts/whatever.sh