All Apps and Add-ons

How do external commands work? Can data be cached?

kcnolan13
Communicator

I want to find out more about how custom commands work in Splunk Apps (specifically for geoip lookup type apps). I've perused the code in several apps now (GeoASN, geoip, SecKit, etc.), and I'm trying to find the most performant way to query a MaxMind Database and map client IP's to Autonomous System Numbers (ASN's).

I keep seeing this kind of thing in each app's transforms.conf file:

[command_name]
external_cmd = command_name.py
fields_list = field1 field2 etc

I'm assuming this is how Splunk knows what data to pipe to which external command... But what I don't know is the real process by which Splunk invokes those commands and passes results back to the eventset.

Here's why I need to know:

If you have to do a lot of MaxMind lookups on a dataset, it's a lot faster if you can cache some results in memory. So, if Splunk is calling out to your add-on application's MaxMind lookup script separately for each lookup, a lot of performance is lost.

So, what I ask is how do external commands like this really work? And what kind of flexibility is there in how they are invoked? Would it be possible to keep a script running so you can cache MaxMind data while you run all the lookups in a streaming-type manner?

As always, thanks for any input you have

1 Solution

aljohnson_splun
Splunk Employee
Splunk Employee

Hi @Kcnolan13!

Awesome question. There are actually a few different types of custom search commands, and, rather than giving a bad attempt at summarizing here, I'll point you towards a great resource - this awesome slide deck from Jacob Leverich at conf2016:

http://conf.splunk.com/files/2016/slides/extending-spl-with-custom-search-commands-and-the-splunk-sd...

Or even better yet, you can listen to the recording of the talk here:

http://conf.splunk.com/files/2016/recordings/extending-spl-with-custom-search-commands-and-the-splun...

^ You can actually implement commands in arbitrary languages (not just python!) using the Chunked External Command Protocol (CEXC). Pretty rad!

I think that should cover everything you're looking to know.

View solution in original post

aljohnson_splun
Splunk Employee
Splunk Employee

Hi @Kcnolan13!

Awesome question. There are actually a few different types of custom search commands, and, rather than giving a bad attempt at summarizing here, I'll point you towards a great resource - this awesome slide deck from Jacob Leverich at conf2016:

http://conf.splunk.com/files/2016/slides/extending-spl-with-custom-search-commands-and-the-splunk-sd...

Or even better yet, you can listen to the recording of the talk here:

http://conf.splunk.com/files/2016/recordings/extending-spl-with-custom-search-commands-and-the-splun...

^ You can actually implement commands in arbitrary languages (not just python!) using the Chunked External Command Protocol (CEXC). Pretty rad!

I think that should cover everything you're looking to know.

aljohnson_splun
Splunk Employee
Splunk Employee

Hi @Kcnolan13 - I noticed you started a second closely-related question here, https://answers.splunk.com/answers/494889/python-sdk-essential-for-custom-commands-protocol.html

Did the answer above answer your original question? If so, please mark the answer as accepted.

0 Karma

kcnolan13
Communicator

My bad -- thought I did that already.

0 Karma
Get Updates on the Splunk Community!

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...