Splunk Dev

How to stream partial results from custom search command in Splunk?

sourav_query_ai
New Member

Hi there, 

I am trying to implement a use case where I have an API that keeps sending partial results (around 50-100) until all the results from the API are done. 

I have implemented a GeneratingCommand for it, and it returns correct results. 

However, I have to wait for quite some time, because Splunk returns results only when all the results from API are collected in Splunk. 

The use case I want: I do not wish to wait for all results, but I want to have the partial results returned in Splunk as soon as they are returned from the API - so I do not have to wait.

I have tried:

1) adding limits.conf

2) using chunked=True

3) editing maxresultrows and maxresults 

4) using flush() results 

5) converting to streaming command and using above steps 

But nothing seems to work. 

Please help, any help would be really appreciated.                

Labels (2)
0 Karma

jakubzak
Explorer

Hi, 

 I've been in exactly the same situation as you and I've ended with splitting my logic into multiple steps. Now I am calling splunk in a loop to get results one by one (there is a bit complicated logic even hidden internally as a custom command not to move huge amount of the data back and forth). Every time I got the data in a loop I propagate it to UI to show it to user and I continue in processing of next record. I was not able to find out any other way how to implement "streaming" so UI user does not have to wait for long time until whole data set is ready.  Hope this will help. 

Get Updates on the Splunk Community!

Accelerate Service Onboarding, Decomposition, Troubleshooting - and more with ITSI’s ...

Accelerate Service Onboarding, Decomposition, Troubleshooting - and more! Faster Time to ValueManaging and ...

New Release | Splunk Enterprise 9.3

Admins and Analyst can benefit from:  Seamlessly route data to your local file system to save on storage ...

2024 Splunk Career Impact Survey | Earn a $20 gift card for participating!

Hear ye, hear ye! The time has come again for Splunk's annual Career Impact Survey!  We need your help by ...