I want to find out more about how custom commands work in Splunk Apps (specifically for geoip lookup type apps). I've perused the code in several apps now (GeoASN, geoip, SecKit, etc.), and I'm trying to find the most performant way to query a MaxMind Database and map client IP's to Autonomous System Numbers (ASN's).
I keep seeing this kind of thing in each app's transforms.conf file:
[command_name]
external_cmd = command_name.py
fields_list = field1 field2 etc
I'm assuming this is how Splunk knows what data to pipe to which external command... But what I don't know is the real process by which Splunk invokes those commands and passes results back to the eventset.
Here's why I need to know:
If you have to do a lot of MaxMind lookups on a dataset, it's a lot faster if you can cache some results in memory. So, if Splunk is calling out to your add-on application's MaxMind lookup script separately for each lookup, a lot of performance is lost.
So, what I ask is how do external commands like this really work? And what kind of flexibility is there in how they are invoked? Would it be possible to keep a script running so you can cache MaxMind data while you run all the lookups in a streaming-type manner?
As always, thanks for any input you have
