topic How to reduce custom search command result chunk size? in Splunk Dev

How to reduce custom search command result chunk size?

joepjisc — Tue, 06 Jun 2023 21:13:58 GMT

We are developing a custom search command to create events, this is using a streaming command with version 2 of the protocol, as the source is quite slow we'd like to send smaller chunks of results back to Splunk than the default 50,000, e.g. chunks of 1,000 events, so that users can view the partial results sooner.

We've tried various approaches including an incrementing integar and calling self.flush() when it is divisable by 1,000, but that caused a buffer full error.

Any suggestions would be really appreciated

... @Configuration(type='streaming') class OurSearchCommand(GeneratingCommand): ... for item in OurGenerator(): item['_time'] = item['timestamp'] yield item

Re: Reduce Custom Search Command Result Chunk Size

DexterMarkley — Tue, 06 Jun 2023 18:14:13 GMT

I know this is an old question, but for anyone else looking for the answer you need to overwrite the record_writer for the class. This is working for me, but I am not sure if there are any other implications of doing this.

self._record_writer._maxresultrows = 1000

Re: Reduce Custom Search Command Result Chunk Size

analyst — Thu, 21 Sep 2023 02:17:29 GMT

@DexterMarkley may you provide the location of file needed to be changes?