How do force a search to finish before invoking a custom search command?

[Marinus]

I'm building a custom search command that performs some visualizations on a dataset outside of Splunk. It has to parse all the search results so it doesn't like chucks of data coming in.

I've set the config to disable streaming, my expectation is that the search will finish and the results will be passed on to my search command. It appears that each chuck of data is executing my search command over and over. Is this a bug? My test rig is running the latest version on Ubuntu 64bit.

[Marinus]

filed! I still think it would be useful if you could via intersplunk detect that you have received the last chunk of event. There must be a simple way to determine it?

[gkanapathy]

And please be aware that with streaming off (and working as intended), you should only be called once, but only with the first 50,000 results, though this can apparently be changed in commands.conf with the maxinputs parameter. I don't know if there aren't other limits from limits.conf that might affect this as well though.

[gkanapathy]

File it as a bug, I mean.

[gkanapathy]

I suspect that there exists a bug such that custom searches are kicked of for every batch of 50,000 events, even if streaming is set off. I suspect that the data is fetched by Splunk, and even passed to the search command, but that the output is simply discarded. Or perhaps the data is fetched, but the search command is invoked without the data input. Would you please file this?

[Marinus]

Thanks for the reply, my search command processes all the rows and produces a file that graphviz parses. It should work like the csv writer in a sense. I can see it's being run more than once since I see multiple temp files being generated by each chunk.

[viz]
streaming = false
filename = viz.py
disabled = 0

Another strategy could be to append and if I could tell that I'm on the last chunk I could invoke my external program.

[Lowell · Accepted solution]

All you should have to do is disable streaming for your search command in commands.conf:

 [yoursearchcommand]
 streaming = false

This should force that your search command gets invoked only once and with your entire dataset. However you may run into new event limits with this approach.

Just to be clear, you will have to restart splunkd for this change to take effect.

If you've already tried this and are still not getting the results you are looking for, please update your question to include some additional details, such as: your full command.conf entry, a skeleton of your code (you can leave out the body if you want). Specifically, are you using the enableheader feature, and if so are you sure that the first argument to splunk.Intersplunk.outputInfo is False?