Thanks Martin! I appreciate you taking the time to look into this with me.
It is good to see that you got similar performance results, but still disappointing that the custom search command is so slow. A couple questions about your comments:
You say that there doesn't seem to be excessive re-launching of python every 255 events, but what led me to that conclusion is the code I inserted to add a "cookie" field, which is a random number computed at the top of each python script. That code might actually be commented out in the ipasn_dat_lookup.py script right now, but you should be able to comment it back in (in that particular script, it might be called the "debug" field, rather than "cookie". sorry for the inconsistency there). I ran the lookup scripts on 1,000,000 events, and stats showed that about (1,000,000/255) unique values for the "cookie" (or "debug") fields existed for the lookup versions of the commands. For the custom search commands, there was only one unique value. So, I concluded that the external lookup scripts were getting re-invoked every 255 events, while the custom search commands were only dispatched once and streamed events in chunks. Would you agree with this line of reasoning?
If the old external lookup method really is faster, I would like to find a definitive answer why.
The best information I could find on the inner workings of traditional Splunk streaming commands was this post from 2012, but it doesn’t explicitly say how much parallelization there is, nor does it say if that applies to external lookup commands. Overall, nothing has explained why my results differ so radically from the performance benchmarks in this powerpoint on the V2 protocol.
Now that you have reviewed my code, note that the MaxMind lookups are being performed the exact same way across both the custom command and external lookup versions. So, with your comment about something "inside the custom command [being] terribly inefficient?" -- do you mean that the custom command protocol itself is tremendously inefficient? If so, is that something we should try to report to the Splunk Dev's? Either way, if external lookups are always going to be this much faster than custom commands, it seems like that should be stated somewhere.
This question as a whole is actually closely related with another I have out there. Since you already have the code, I was hoping you might be able to take a look at this:
Try as I may, that custom command refuses to distribute to indexers. I feel like an answer to that question may be more achievable, and I would very much like to find out what the problem is. I have implemented versions with the V2 protocol and the old Intersplunk way of doing things, and the streaming command isn't showing up in remoteSearch either way.
... View more