topic Re: How do you send events from a custom command asynchronously without waiting for the whole command to finish? in Splunk Search

How do you send events from a custom command asynchronously without waiting for the whole command to finish?

yogevyuval — Mon, 05 Nov 2018 15:53:07 GMT

Hi,

I have a custom generating command that queries an external API and yields the results as events.

As the API gives me the results in chunks and can be slow sometimes, I would like to be able to see the yielded events that I got so far in the Splunk interface, and have incoming events continue to be appended — Just like when I search a big index, and the events seems to be "streamed" to Splunk when they are ready while the search is working. Then, I can see some of the results before the entire search is ready.

Currently, what is happening is that I see no results until the command finishes entirely, and then I see all of the events at once.
If my API sends me 100 events over 10 seconds, I would like to see some of the events immediately.

I thought that this happens automatically when I use the generator and yield pattern, but it seems that it's not the case.

Re: How do you send events from a custom command asynchronously without waiting for the whole command to finish?

sduff_splunk — Tue, 06 Nov 2018 01:30:08 GMT

When you created your custom command, did you specify that it should be a streaming command?
http://dev.splunk.com/view/python-sdk/SP-CAAAEU2#streamingcommand
http://docs.splunk.com/DocumentationStatic/PythonSDK/1.6.5/searchcommands.html#splunklib.searchcommands.StreamingCommand

Re: How do you send events from a custom command asynchronously without waiting for the whole command to finish?

yogevyuval — Tue, 06 Nov 2018 05:48:47 GMT

Yes, I did set this to be a streaming command.

I noticed this in the docs you sent :Streaming commands typically filter, augment, or update, search result records. Splunk will send them in batches of up to 50,000 records.

Does this mean that I will only see the first batch after yielding 50,000 events? If so, how can I change that number to be much less than that? (~100) @sduff