topic Re: How to custom search command (v2/chunked) protocol documentation? in Splunk Dev

How to custom search command (v2/chunked) protocol documentation?

spunk_enthusias — Thu, 30 Mar 2023 18:44:03 GMT

The page About non-Python custom search commands mentions that it is possible to write v2 custom search commands in languages other than Python, but there is absolutely no information about how such a thing would be implemented. What's the protocol?

The closest thing to an explanation of the protocol I've found is NDietrich's GitHub repo, and their accompanying talk which I find rather disappointing.

How come there is no official information to be found about it?

Re: Custom search command (v2/chunked) protocol documentation

spunk_enthusias — Thu, 30 Mar 2023 17:07:53 GMT

Holy crap guys, I found some hints in the Splunk Dev For All app!!! This has a fairly small (Python 2) utility library called "cexec" implementing the chunked search protocol! Here's a part of its docstring:

This library abstracts away some of the low-level details of writing
"chunked" custom search commands for Splunk (e.g. byte-level protocol
parsing). However, it still requires a fair bit of background on how
the chunked protocol works at a semantic level. For a detailed
description of the protocol, read:
https://confluence.splunk.com/display/PROD/Chunked+External+Command+Protocol+v1.0
At a high-level, the Splunk search pipeline operates on "chunks" of
search results. Thus, when a "chunked" custom search command is in a
search pipeline, Splunk will send chunks to the external command (on
stdin) and expect chunks in reply (on stdout).
This library implements a BaseChunkHandler class that handles most of
the details of receiving and sending chunks. Developers are expected
to extend this class with their own handler() method to actually do
useful work on search results.

Sadly the linked Confluence page is offline, not saved by the Internet Archive and a web search for "Chunked External Command Protocol v1.0" yields exactly 0 results (how often does that happen?).

Luckily the library only has 368 lines, is well commented and quite readable!

So that's probably some of the best documentation we have. Still baffling that Splunk Inc. seems to want to keep information about the chunked search protocol a secret. That together with the atrocious performance (200ms MINIMUM) makes using them for utility functionality inviable. And do you know what that means? We'll use the custom command exactly once and do all processing outside of Splunk.

If the bad experience of custom search commands is meant to hamper migration away, congratulations, you played yourselves.

Re: How to custom search command (v2/chunked) protocol documentation?

thellmann — Wed, 26 Apr 2023 21:25:57 GMT

We recently added a page to the Custom Search Commands manual on dev.splunk.com that might have some of the information you're looking for: https://dev.splunk.com/enterprise/docs/devtools/customsearchcommands/nonpythonscscs

Re: How to custom search command (v2/chunked) protocol documentation?

spunk_enthusias — Fri, 12 May 2023 12:34:29 GMT

Thank you, but ... what do I do with that?

Say I want to implement a command in Java. The page has a helpful side note on the java path, but the table says to implement a command in Python I have to use ... The Splunk SDK for Python? And the guide just stops at how to point Splunk to my application, which is like 5% of the way there.

Do you expect people to know the protocol already? Because again, I don't see ANY documentation on it and I don't find the Python source to be all that readable either, though arguably that *is* the best documentation I've seen.

A hint on how the DB Connect folks did it would be helpful. Though then again they also use a ton of shims until they get to launch their java app, but at least the protocol appears to be implemented there.