topic Re: How can I use reduce to avoid previewing a report from a slow search until it's done? in Splunk Search

How can I use reduce to avoid previewing a report from a slow search until it's done?

wegscd — Tue, 29 Sep 2020 15:29:07 GMT

I have a ReportingCommand written in Python, and the SPL that feeds it is slowish.

To minimize visual churn on the screen, I don't want my command to provide intermediate results: I just want the reduce() method to get run once when all the data is ready to process.

I am using SCP 2, and I have my command annotated with run_in_preview=False, but reduce() is still getting run multiple times (with more and more input records, and preview=True is in self.metadata).

The SPL:

sourcetype=whr:sap:abap:audit | stats count, values(sourcetype) as sourcetype by sid | append [ | inputlookup whirlpool_sap_instances ] | sapabapsummarize

The command

@Configuration(requires_preop=True, run_in_preview=False)
class SapAbapSummarizeCommand(ReportingCommand):

    @Configuration()
    def map(self, records):
        self.logger.info('SapAbapSummarizeCommand.map, phase = %s', self.phase)
        r_count = 0
        for record in records:
            r_count = r_count + 1
            yield record
        self.logger.info('SapAbapSummarizeCommand.map done: %d', r_count)

    @Configuration(run_in_preview=False)
    def reduce(self, records):
        self.logger.info('SapAbapSummarizeCommand.reduce, phase = %s', self.phase)
        self.logger.info('SapAbapSummarizeCommand.reduce, metadata = %s', self.metadata)
        r_count = 0

        for record in records:
            r_count = r_count + 1
            # (other processing)

        self.logger.info('SapAbapSummarizeCommand.reduce done: %d', r_count)

commands.conf:

[sapabapsummarize]
filename = sap_abap_summarize.py
chunked = true

How do I get reduce to know that all the results are in? The 'Preview' flag in the metadata is always True.

Re: How can I use reduce to avoid previewing a report from a slow search until it's done?

jkat54 — Tue, 29 Sep 2020 15:29:16 GMT

Put this in commands.conf

[yourcommand]
run_in_preview = false

Which will disable preview

Re: How can I use reduce to avoid previewing a report from a slow search until it's done?

wegscd — Tue, 29 Sep 2020 15:29:22 GMT

That is the SCP V1 version of SCP V2's @Configuration(run_in_preview=False), which is already been done.

The extra runs I am seeing are during preview.

Re: How can I use reduce to avoid previewing a report from a slow search until it's done?

jkat54 — Wed, 23 Aug 2017 14:24:30 GMT

You're clearing the configuration prior to map though... is that the cause?

Re: How can I use reduce to avoid previewing a report from a slow search until it's done?

wegscd — Tue, 29 Sep 2020 15:29:37 GMT

not sure, but I need to clear it out. If I put run_in_preview=False in map, I get

08-23-2017 10:29:38.001 INFO  ChunkedExternProcessor - Running process: /opt/splunk/bin/python /opt/splunk/etc/apps/wegscd_command_test/bin/sap_abap_summarize.py
08-23-2017 10:29:38.104 WARN  ChunkedExternProcessor - Unsupported metadata field sent by custom search command: run_in_preview
08-23-2017 10:29:38.104 INFO  SearchParser - PARSING: sapabapsummarize phase="map"

Re: How can I use reduce to avoid previewing a report from a slow search until it's done?

jkat54 — Wed, 23 Aug 2017 15:04:25 GMT

Interesting the whole v2 is new to me. What if you do it in commands.conf anyways?

Just curious.

Re: How can I use reduce to avoid previewing a report from a slow search until it's done?

wegscd — Wed, 23 Aug 2017 17:40:34 GMT

It seems to behave using scp v1. I am putting together a test case; I'm getting very strange results.....

Re: How can I use reduce to avoid previewing a report from a slow search until it's done?

jbjerke_splunk — Mon, 19 Nov 2018 15:21:38 GMT

Hi wegscd

I'm having the same issue. Did you get anywhere with this? Did you managed to prevent previews somehow?