It appears that this problem arises due to the user not having the right splunk "capability". I'm not sure which capability is required but when I added the "admin" role to my user, the problem went away.
I would love to know specifically which capability is required so I can make my user minimally enabled to use runshellscript.py
I am also having this problem. I have had a working script in the /opt/splunk/bin/scripts/ directory for several months now, and suddenly it is not firing when my alerts are generated. I checked the splunkd.log and found command="runshellscript" cannot find script at /opt/splunk/bin/scripts/.py.
Anyone know what might cause this behavior? The splunk instance has not be upgraded or altered in many months, so the files shouldn't be changing or randomly fail to find something that has existed for that long.