Splunk Search

How do you send events from a custom command asynchronously without waiting for the whole command to finish?

yogevyuval
Explorer

Hi,

I have a custom generating command that queries an external API and yields the results as events.

As the API gives me the results in chunks and can be slow sometimes, I would like to be able to see the yielded events that I got so far in the Splunk interface, and have incoming events continue to be appended — Just like when I search a big index, and the events seems to be "streamed" to Splunk when they are ready while the search is working. Then, I can see some of the results before the entire search is ready.

Currently, what is happening is that I see no results until the command finishes entirely, and then I see all of the events at once.
If my API sends me 100 events over 10 seconds, I would like to see some of the events immediately.

I thought that this happens automatically when I use the generator and yield pattern, but it seems that it's not the case.

sduff_splunk
Splunk Employee
Splunk Employee
0 Karma

yogevyuval
Explorer

Yes, I did set this to be a streaming command.

I noticed this in the docs you sent :Streaming commands typically filter, augment, or update, search result records. Splunk will send them in batches of up to 50,000 records.

Does this mean that I will only see the first batch after yielding 50,000 events? If so, how can I change that number to be much less than that? (~100) @sduff

0 Karma
Get Updates on the Splunk Community!

Splunk Observability Cloud's AI Assistant in Action Series: Auditing Compliance and ...

This is the third post in the Splunk Observability Cloud’s AI Assistant in Action series that digs into how to ...

Splunk Community Badges!

  Hey everyone! Ready to earn some serious bragging rights in the community? Along with our existing badges ...

What You Read The Most: Splunk Lantern’s Most Popular Articles!

Splunk Lantern is a Splunk customer success center that provides advice from Splunk experts on valuable data ...