I built a simple app with the add-on builder (using the python script inputs) and then copied it in the "master-apps" of my cluster master.
After successful bundle replication I see the app is on the indexers as well, but I noticed some strange things immediately:
I already checked app sizes on Master and Peers and it is the same everywhere. No files are missing from the app...
Any ideas why this might happen? Some strange limitations on "slave-apps" content maybe?
Ok, I have a workaround that seems to solve the issue...but it's dirty.
The root cause is that splunkd does not search for any scripts,icons,css,whatever resources when apps are under "slave-apps". This is how it works currently, I don't know why. If you create a symbolic link with the same name under "apps" directory the problem will disappear.
E.g. you have an app, called "TA-myapp" and you deploy it through the Cluster Master to "slave-apps" on the Cluster Peers. The app path would be something like "/opt/splunk/etc/slave-apps/TA-myapp". In order to solve the issue with python scripts not found, simply run the command:
ln -s /opt/splunk/etc/slave-apps/TA-myapp /opt/splunk/etc/apps/TA-myapp
This will create a symbolic link under "/opt/splunk/etc/apps" that points to the app under "slave-apps" and then Splunkd would be able to find the missing resource. This will fix also issues with missing icons (e.g. like in the case of some other add-ons like TA for PaloAlto).
Don't deploy scripted inputs to indexer clusters. Each indexer will run the script(s) and you'll end up with duplicated data.
Instead, deploy the app to a heavy forwarder.
Yes, it is related to the issue you are having. If you weren't trying to do something that shouldn't be done, you wouldn't have the issue.
I am trying to run scripts on all the indexers in order to do some housekeeping locally on the machine. The resulting output is collected as events, which are relatable only to the local machine, e.g. they are not "duplicate" in any sense whatsoever. Kinda like the stuff in "_internal", you know?
If the peers weren't in a cluster I would've used the Deployment Server and probably it would've worked, but since the peers are part of an indexer cluster...guess I'm stuck with the master-apps method.
Which gives me the issue...
What you are talking about is perfectly fine, but doesn't really explain why it doesn't/shouldn't work in my case.
Ok, thanks for the feedback, although it isn't related to the issue, I'm afraid.
I think I found something else though... The permissions look very different when doing "ls -al" on master-apps compared to the slave-apps... Maybe that's the reason.