In fact, I have asked that very question in 2017, did not get an answer and created this solution. Here is the link to my post: Solved: Re: Custom search command called multiple times - Splunk Community There, you can find a more detailed description of my solution. ... View more

Here I am, over 3 years later, finding my own answer to help me out again 🙂  ... View more

  It seems like your custom search command is being executed twice because of the way Splunk handles the map/reduce phases in a distributed search environment. This behavior is expected in certain scenarios, and it's often related to how Splunk distributes and processes data across different search peers. In your specific case, the key information is found in the search.log: 09-13-2018 11:33:08.462 INFO ParallelReducePolicy - Current Search Head doesn't have any usable peers to use. 09-13-2018 11:33:08.462 INFO PhaseNodeGenerationVisitor - User lacking run_multi_phased_searches, rolling back to 2-phase mode. Splunk is running the map phase twice because it's not able to parallelize the search across multiple search peers. As a result, it falls back to a two-phase execution (map and reduce) on the search head itself. To avoid this behavior, you might want to consider the following: Parallelization: Ensure that your search environment has multiple search peers that can be utilized for parallel processing. This may involve configuring search peer clustering or making adjustments to your distributed environment. run_multi_phased_searches: The log mentions that the user lacks the run_multi_phased_searches capability. This capability allows Splunk to run multiple phases on different search peers simultaneously. You might want to check the user's role and capabilities to ensure it has the necessary permissions. Debugging: Consider adding more logging statements to your custom search command to trace the execution flow and see if it provides additional insights into why it's being called twice. Splunk Documentation: Check the Splunk documentation for custom search commands and distributed search to see if there are any specific guidelines or recommendations for handling distributed environments. Splunk Answers: Search the Splunk Answers community for similar issues or ask a question there. The community is active, and you might find someone who has encountered and resolved a similar problem. Keep in mind that Splunk's distributed search behavior can be complex, and understanding how it distributes and processes data is crucial for developing efficient custom search commands.   I hope this will help Thanks rasad4468   ... View more

When running   | inputlookup testlookup   (which is an external lookup) I get the error message: The lookup table 'testlookup' requires a .csv or KV store lookup definition ... so I assume this isn't an intended use case. Quite a bummer because (as per some of my earlier posts) custom search commands kind of suck. ... View more

Tabling works, but that's not enough if you need to carry along hidden values for Drilldowns. Tokens "kind of" work with the <fields> tag, but they don't seem to update when the token changes. Classic Splunk W. ... View more

Thank you, but ... what do I do with that? Say I want to implement a command in Java. The page has a helpful side note on the java path, but the table says to implement a command in Python I have to use ... The Splunk SDK for Python? And the guide just stops at how to point Splunk to my application, which is like 5% of the way there. Do you expect people to know the protocol already? Because again, I don't see ANY documentation on it and I don't find the Python source to be all that readable either, though arguably that *is* the best documentation I've seen.  A hint on how the DB Connect folks did it would be helpful. Though then again they also use a ton of shims until they get to launch their java app, but at least the protocol appears to be implemented there. ... View more

What a crazy oversight, right? This isn't just some authenticated port that you can't do anything with or something someone would ever actually normally connect to from the outside. No, this thing opens up a very direct tunnel into the java server to the whole network using the Splunk custom command protocol. What?!?! Splunk has been negatively impressing me regarding everything surrounding custom search commands (the DB Connect commands are implemented as such) and DB Connect is an especially hard-to-predict-hard-to-debug example. While I can't answer your question directly, I can tell you what didn't work for me. I was hoping that setting the vmopts option -Daddress=127.0.0.1 or -Dserver.address=127.0.0.1 would help, since that's what works for some Spring Boot applications (which I'm not sure DB Connect is, but it might be). Perhaps firewalling would work or network namespaces/containerization. ... View more

This doesn't seem to work for clearing the scripts cache ($SPLUNK_API/servicesNS/-/-/admin/script/_reload). It's not equivalent to $SPLUNK_WEB/en-US/debug/refresh?entity=admin/script. ... View more

I'm looking for this as well, for colocation reasons. But so far I haven't found a way. For people playing along at home: web.conf has a few options for dashboards (like dashboard_html_allow_embeddable_content), but seemingly nothing to allow embedding scripts. ... View more

Is there really no better way than this? ... View more

Now here's an interesting tidbit: Where a bare-bones Python script using Splunklib takes 170ms to dispatch.evaluate, a dbxquery from DB Connect takes 80ms for a whole query! And that's also with a chunked python3 search command. So what's different? It doesn't use splunklib! Or at least not the Python version. The Python script acting as the bridge simply forwards the search protocol messages to a TCP server which likely uses the Java implementation. I've said before that splunklib could use some attention - case in point. ... View more

Despite our use cases being slightly different I think we want the same thing: Some form of persistence rather than the file being called over and over. I ask the question a bit more generally here: https://community.splunk.com/t5/Building-for-the-Splunk-Platform/Custom-Search-Command-V2-Chunked-Persist-script/m-p/626063#M10884 ... View more

I wouldn't consider this problem solved. It remains annoyingly hard to run custom search commands from the command line because the protocol is entirely undocumented and no tooling is available. ... View more

I also have this issue with a chunked custom search command. I simply call the command in a search (Fast/Verbose doesn't make a difference) and the command is called twice. Preview is off. The job inspector shows 2 invocations, but doesn't explain them. My logs also show 2 invocations, with some differences: The metadata field is a bit shorter on the first invocation Only in the second invocation is the search_results_info field filled (with a lot of general information). Interestingly both times the action is getinfo, preview is False and streaming_command_will_restart is True. That getinfo is the first call seems to be a thing of the V2-Protocol from my reading of the Splunklib code. Edit: At some point I get this message:   ERROR ChunkedExternProcessor [27114 ChunkedExternProcessorStderrLogger] - stderr: Exception ignored in: <_io.TextIOWrapper name='<stdout>' mode='w' encoding='UTF-8'>   It  should be noted that even though the file is loaded twice and the prepare method is called twice, the generate method is not. Too bad that for my application loading the file is exactly the problem. ... View more

Ha, yep, I had to guess this myself. The custom search commands (or splunklib in general) docs could use some work. ... View more

Thank you! This may come in handy if I don't manage to find another option to keep the search command's process alive and end up having to manage one myself. ... View more

[expose:all] methods = GET,POST pattern = ** skipCSRFProtection = 1 I'm trying to disable CSRF protection for other reasons and can confirm that this approach doesn't work for me. ... View more

This solves all my "why the hell isn't it adding this field no matter what I do" related problems. My previous workaround was doing a makeresults and doing an addcols to it, which is of course terrible. Why does Splunk behave like this by default? ... View more