I created .sh scripts that do the following:
#!/bin/bash
# Name of the service to monitor
SERVICE_NAME="tomcat9"
# Check if the service is running
SERVICE_STATUS=$(systemctl is-active "$SERVICE_NAME.service")
# Output status for Splunk
if [ "$SERVICE_STATUS" == "active" ]; then
echo "$(date): Service $SERVICE_NAME is running."
else
echo "$(date): Service $SERVICE_NAME is NOT running."
fi
The above is obviously what Im using for Tomcat but I have others all doing the thing just different service names. These scripts reside in:
/opt/splunkforwarder/bin/scripts
Additionally I have configured these scripts to be run in /opt/splunkforwarder/etc/system/local/inputs.conf an example of what that looks like is below:
[script:///opt/splunkforwarder/bin/scripts/monitor_service_<service_name>.sh]
disabled = false
interval = 60
index = services
sourcetype = service_status
As you can see I also have configured the following:
index = services
sourcetype = service_status
These are also configured in Splunk Enterprise respectively and the index is configured for Search, in linux Splunk is the owner and the group is also Splunk. Additionally all of the scripts are executable and successfully run when I test them, however none of this data seems to be passed from the forwarder as none of the expected data is returned including the recognition of the index and sourcetype in Search. Additionally I have attached a screen capture of splunkd.log showing the scripts as being recognized.
