Is there any way to run scripted inputs with commands in them that aren't installed on the splunksearch instance?

[bsg273]

I have created a scripted input (/opt/splunk/etc/apps/mytestapp/bin/scriptedinput1.sh) to run against my kubetools instances that simply runs a kubectl command and outputs json:

#!/bin/sh

/usr/bin/kubectl -n mynamespace1 get deployments,statefulsets -o json

However, after I go to set up the scripted input in the Data inputs section of the Splunk console and run a search, I'm seeing this error in splunkd.log:

ERROR ExecProcessor - message from "/opt/splunk/etc/apps/mytestapp/bin/scriptedinput1.sh" /opt/splunk/etc/apps/mytestapp/bin/scriptedinput1.sh: line 3: /usr/bin/kubectl: No such file or directory

I suspect it's throwing this error because kubetools is only installed on the kubetools instances and not the splunksearch instance.  Is there any way to run scripted inputs with commands in them that aren't installed on the splunksearch instance?  If not, what alternative solution would be recommended?  Any assistance with solving this would be greatly appreciated.

[richgalloway]

Install a Universal Forwarder on the remote instance and run the scripted input there.

[bsg273]

So I already had Splunk Forwarder installed on the remote instance (my-kubetools-1), so created the script (/opt/splunkforwarder/etc/apps/Splunk_TA_nix/bin/scriptedinput1.sh) on there, gave it 700 permissions and changed its ownership to splunk:splunk.  I also appended the following to the end of my inputs.conf (/opt/splunkforwarder/etc/apps/Splunk_TA_nix/local/inputs.conf):

[script://./bin/scriptedinput1.sh]
disabled = 0
interval = 60
sourcetype = scriptedinput1
index = om

However, I'm still not getting any events when running the following search:

host=my-kubetools-1* index=om sourcetype=scriptedinput1

 Did I miss something?

[richgalloway]

Verify the script runs correctly and writes output to stdout.

Be sure to restart the UF after changing .conf files.

Verify you can search for the UF's internal logs (host=my-kubetools-1* index=_internal).  If you can't then you have a connectivity problem and that needs to be fixed before data can be indexed.