Splunk Search
Highlighted

How do you send events from a custom command asynchronously without waiting for the whole command to finish?

Engager

Hi,

I have a custom generating command that queries an external API and yields the results as events.

As the API gives me the results in chunks and can be slow sometimes, I would like to be able to see the yielded events that I got so far in the Splunk interface, and have incoming events continue to be appended — Just like when I search a big index, and the events seems to be "streamed" to Splunk when they are ready while the search is working. Then, I can see some of the results before the entire search is ready.

Currently, what is happening is that I see no results until the command finishes entirely, and then I see all of the events at once.
If my API sends me 100 events over 10 seconds, I would like to see some of the events immediately.

I thought that this happens automatically when I use the generator and yield pattern, but it seems that it's not the case.

0 Karma
Highlighted

Re: How do you send events from a custom command asynchronously without waiting for the whole command to finish?

Splunk Employee
Splunk Employee
0 Karma
Highlighted

Re: How do you send events from a custom command asynchronously without waiting for the whole command to finish?

Engager

Yes, I did set this to be a streaming command.

I noticed this in the docs you sent :Streaming commands typically filter, augment, or update, search result records. Splunk will send them in batches of up to 50,000 records.

Does this mean that I will only see the first batch after yielding 50,000 events? If so, how can I change that number to be much less than that? (~100) @sduff

0 Karma