Splunk Search

How do you send events from a custom command asynchronously without waiting for the whole command to finish?

yogevyuval
Explorer

Hi,

I have a custom generating command that queries an external API and yields the results as events.

As the API gives me the results in chunks and can be slow sometimes, I would like to be able to see the yielded events that I got so far in the Splunk interface, and have incoming events continue to be appended — Just like when I search a big index, and the events seems to be "streamed" to Splunk when they are ready while the search is working. Then, I can see some of the results before the entire search is ready.

Currently, what is happening is that I see no results until the command finishes entirely, and then I see all of the events at once.
If my API sends me 100 events over 10 seconds, I would like to see some of the events immediately.

I thought that this happens automatically when I use the generator and yield pattern, but it seems that it's not the case.

sduff_splunk
Splunk Employee
Splunk Employee
0 Karma

yogevyuval
Explorer

Yes, I did set this to be a streaming command.

I noticed this in the docs you sent :Streaming commands typically filter, augment, or update, search result records. Splunk will send them in batches of up to 50,000 records.

Does this mean that I will only see the first batch after yielding 50,000 events? If so, how can I change that number to be much less than that? (~100) @sduff

0 Karma
Get Updates on the Splunk Community!

Routing Data to Different Splunk Indexes in the OpenTelemetry Collector

This blog post is part of an ongoing series on OpenTelemetry. The OpenTelemetry project is the second largest ...

Getting Started with AIOps: Event Correlation Basics and Alert Storm Detection in ...

Getting Started with AIOps:Event Correlation Basics and Alert Storm Detection in Splunk IT Service ...

Register to Attend BSides SPL 2022 - It's all Happening October 18!

Join like-minded individuals for technical sessions on everything Splunk!  This is a community-led and run ...