Splunk Search

How do you send events from a custom command asynchronously without waiting for the whole command to finish?

yogevyuval
Explorer

Hi,

I have a custom generating command that queries an external API and yields the results as events.

As the API gives me the results in chunks and can be slow sometimes, I would like to be able to see the yielded events that I got so far in the Splunk interface, and have incoming events continue to be appended — Just like when I search a big index, and the events seems to be "streamed" to Splunk when they are ready while the search is working. Then, I can see some of the results before the entire search is ready.

Currently, what is happening is that I see no results until the command finishes entirely, and then I see all of the events at once.
If my API sends me 100 events over 10 seconds, I would like to see some of the events immediately.

I thought that this happens automatically when I use the generator and yield pattern, but it seems that it's not the case.

sduff_splunk
Splunk Employee
Splunk Employee
0 Karma

yogevyuval
Explorer

Yes, I did set this to be a streaming command.

I noticed this in the docs you sent :Streaming commands typically filter, augment, or update, search result records. Splunk will send them in batches of up to 50,000 records.

Does this mean that I will only see the first batch after yielding 50,000 events? If so, how can I change that number to be much less than that? (~100) @sduff

0 Karma
Get Updates on the Splunk Community!

Welcome to the Splunk Community!

(view in My Videos) We're so glad you're here! The Splunk Community is place to connect, learn, give back, and ...

Tech Talk | Elevating Digital Service Excellence: The Synergy of Splunk RUM & APM

Elevating Digital Service Excellence: The Synergy of Real User Monitoring and Application Performance ...

Adoption of RUM and APM at Splunk

    Unleash the power of Splunk Observability   Watch Now In this can't miss Tech Talk! The Splunk Growth ...