I have created a custom search command, using the streaming search templates provided for the splunk SDK.
It is a simple "take in the results from x field, manipulate, and add in a couple new fields". This runs fine on my single instance development server.
When I push it out to my search head cluster, I have 2 problems:
1) (Coincidence?) My Indexers all spiked in CPU around the same time i ran my search. Can a search head custom search impact the indexer? Looking at the docs, it seems like it would only impact the search head.
2) My search runs fine in a standalone instance, but in a distributed instance (SHC, Index cluster), I get this error. I dont see any more info on this -- how can I debug something so vague in splunk?
[idx01-g,idx01-k,idx02-g,idx02-k,idx03-g,idx03-k,idx04-g,idx04-k,idx05-g,idx05-k] Streamed search execute failed because: Error in 'punycode' command: External search command exited unexpectedly with non-zero error code 1.
Further investigation throws this (not on standalone, SHC only) but I changed it and am still getting this error.
Hold up. None of those errors seem applicable, because:
Further reading shows me that streaming searches can happen on the indexers AND/OR the search heads. Why! How? And more importantly, which one should I use?
I ran a "sort" in front of my custom command |sort -url | punycode fieldname=url and I believe this forces my command to run locally. After doing this, my command works.
So, I do not understand if this needs to be run on indexers and SH, or just SH.
And...
If it needs to run on indexers, do I just install my app on them also?