We are developing a custom search command to create events, this is using a streaming command with version 2 of the protocol, as the source is quite slow we'd like to send smaller chunks of results back to Splunk than the default 50,000, e.g. chunks of 1,000 events, so that users can view the partial results sooner.
We've tried various approaches including an incrementing integar and calling self.flush() when it is divisable by 1,000, but that caused a buffer full error.
Any suggestions would be really appreciated
...
@Configuration(type='streaming')
class OurSearchCommand(GeneratingCommand):
...
for item in OurGenerator():
item['_time'] = item['timestamp']
yield item
