All Apps and Add-ons
Highlighted

How do external commands work? Can data be cached?

Communicator

I want to find out more about how custom commands work in Splunk Apps (specifically for geoip lookup type apps). I've perused the code in several apps now (GeoASN, geoip, SecKit, etc.), and I'm trying to find the most performant way to query a MaxMind Database and map client IP's to Autonomous System Numbers (ASN's).

I keep seeing this kind of thing in each app's transforms.conf file:

[command_name]
external_cmd = command_name.py
fields_list = field1 field2 etc

I'm assuming this is how Splunk knows what data to pipe to which external command... But what I don't know is the real process by which Splunk invokes those commands and passes results back to the eventset.

Here's why I need to know:

If you have to do a lot of MaxMind lookups on a dataset, it's a lot faster if you can cache some results in memory. So, if Splunk is calling out to your add-on application's MaxMind lookup script separately for each lookup, a lot of performance is lost.

So, what I ask is how do external commands like this really work? And what kind of flexibility is there in how they are invoked? Would it be possible to keep a script running so you can cache MaxMind data while you run all the lookups in a streaming-type manner?

As always, thanks for any input you have

Highlighted

Re: How do external commands work? Can data be cached?

Splunk Employee
Splunk Employee

Hi @Kcnolan13!

Awesome question. There are actually a few different types of custom search commands, and, rather than giving a bad attempt at summarizing here, I'll point you towards a great resource - this awesome slide deck from Jacob Leverich at conf2016:

http://conf.splunk.com/files/2016/slides/extending-spl-with-custom-search-commands-and-the-splunk-sd...

Or even better yet, you can listen to the recording of the talk here:

http://conf.splunk.com/files/2016/recordings/extending-spl-with-custom-search-commands-and-the-splun...

^ You can actually implement commands in arbitrary languages (not just python!) using the Chunked External Command Protocol (CEXC). Pretty rad!

I think that should cover everything you're looking to know.

View solution in original post

Highlighted

Re: How do external commands work? Can data be cached?

Splunk Employee
Splunk Employee

Hi @Kcnolan13 - I noticed you started a second closely-related question here, https://answers.splunk.com/answers/494889/python-sdk-essential-for-custom-commands-protocol.html

Did the answer above answer your original question? If so, please mark the answer as accepted.

0 Karma
Highlighted

Re: How do external commands work? Can data be cached?

Communicator

My bad -- thought I did that already.

0 Karma