All Apps and Add-ons

How do external commands work? Can data be cached?

kcnolan13
Communicator

I want to find out more about how custom commands work in Splunk Apps (specifically for geoip lookup type apps). I've perused the code in several apps now (GeoASN, geoip, SecKit, etc.), and I'm trying to find the most performant way to query a MaxMind Database and map client IP's to Autonomous System Numbers (ASN's).

I keep seeing this kind of thing in each app's transforms.conf file:

[command_name]
external_cmd = command_name.py
fields_list = field1 field2 etc

I'm assuming this is how Splunk knows what data to pipe to which external command... But what I don't know is the real process by which Splunk invokes those commands and passes results back to the eventset.

Here's why I need to know:

If you have to do a lot of MaxMind lookups on a dataset, it's a lot faster if you can cache some results in memory. So, if Splunk is calling out to your add-on application's MaxMind lookup script separately for each lookup, a lot of performance is lost.

So, what I ask is how do external commands like this really work? And what kind of flexibility is there in how they are invoked? Would it be possible to keep a script running so you can cache MaxMind data while you run all the lookups in a streaming-type manner?

As always, thanks for any input you have

1 Solution

aljohnson_splun
Splunk Employee
Splunk Employee

Hi @Kcnolan13!

Awesome question. There are actually a few different types of custom search commands, and, rather than giving a bad attempt at summarizing here, I'll point you towards a great resource - this awesome slide deck from Jacob Leverich at conf2016:

http://conf.splunk.com/files/2016/slides/extending-spl-with-custom-search-commands-and-the-splunk-sd...

Or even better yet, you can listen to the recording of the talk here:

http://conf.splunk.com/files/2016/recordings/extending-spl-with-custom-search-commands-and-the-splun...

^ You can actually implement commands in arbitrary languages (not just python!) using the Chunked External Command Protocol (CEXC). Pretty rad!

I think that should cover everything you're looking to know.

View solution in original post

aljohnson_splun
Splunk Employee
Splunk Employee

Hi @Kcnolan13!

Awesome question. There are actually a few different types of custom search commands, and, rather than giving a bad attempt at summarizing here, I'll point you towards a great resource - this awesome slide deck from Jacob Leverich at conf2016:

http://conf.splunk.com/files/2016/slides/extending-spl-with-custom-search-commands-and-the-splunk-sd...

Or even better yet, you can listen to the recording of the talk here:

http://conf.splunk.com/files/2016/recordings/extending-spl-with-custom-search-commands-and-the-splun...

^ You can actually implement commands in arbitrary languages (not just python!) using the Chunked External Command Protocol (CEXC). Pretty rad!

I think that should cover everything you're looking to know.

aljohnson_splun
Splunk Employee
Splunk Employee

Hi @Kcnolan13 - I noticed you started a second closely-related question here, https://answers.splunk.com/answers/494889/python-sdk-essential-for-custom-commands-protocol.html

Did the answer above answer your original question? If so, please mark the answer as accepted.

0 Karma

kcnolan13
Communicator

My bad -- thought I did that already.

0 Karma
Get Updates on the Splunk Community!

Splunk Observability Cloud | Unified Identity - Now Available for Existing Splunk ...

Raise your hand if you’ve already forgotten your username or password when logging into an account. (We can’t ...

Index This | How many sides does a circle have?

February 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

Registration for Splunk University is Now Open!

Are you ready for an adventure in learning?   Brace yourselves because Splunk University is back, and it's ...