Custom Search Fails to run in distributed environment.

[oliverja]

I have created a custom search command, using the streaming search templates provided for the splunk SDK. 

It is a simple "take in the results from x field, manipulate, and add in a couple new fields". This runs fine on my single instance development server.

When I push it out to my search head cluster, I have 2 problems:

1) (Coincidence?) My Indexers all spiked in CPU around the same time i ran my search. Can a search head custom search impact the indexer? Looking at the docs, it seems like it would only impact the search head.

2) My search runs fine in a standalone instance, but in a distributed instance (SHC, Index cluster), I get this error. I dont see any more info on this -- how can I debug something so vague in splunk? 

[idx01-g,idx01-k,idx02-g,idx02-k,idx03-g,idx03-k,idx04-g,idx04-k,idx05-g,idx05-k] Streamed search execute failed because: Error in 'punycode' command: External search command exited unexpectedly with non-zero error code 1.

oliverja_1-1656061135354.png

Further investigation throws this (not on standalone, SHC only) but I changed it and am still getting this error.

oliverja_0-1656061117947.png

[oliverja]

Hold up. None of those errors seem applicable, because:

Further reading shows me that streaming searches can happen on the indexers AND/OR the search heads. Why! How? And more importantly, which one should I use?

I ran a "sort" in front of my custom command  |sort -url | punycode fieldname=url and I believe this forces my command to run locally. After doing this, my command works.

So, I do not understand if this needs to be run on indexers and SH, or just SH. 

And...

If it needs to run on indexers, do I just install my app on them also?