Splunk Search

How can I use reduce to avoid previewing a report from a slow search until it's done?

wegscd
Contributor

I have a ReportingCommand written in Python, and the SPL that feeds it is slowish.

To minimize visual churn on the screen, I don't want my command to provide intermediate results: I just want the reduce() method to get run once when all the data is ready to process.

I am using SCP 2, and I have my command annotated with run_in_preview=False, but reduce() is still getting run multiple times (with more and more input records, and preview=True is in self.metadata).

The SPL:

sourcetype=whr:sap:abap:audit | stats count, values(sourcetype) as sourcetype by sid | append [ | inputlookup whirlpool_sap_instances ] | sapabapsummarize

The command

@Configuration(requires_preop=True, run_in_preview=False)
class SapAbapSummarizeCommand(ReportingCommand):

    @Configuration()
    def map(self, records):
        self.logger.info('SapAbapSummarizeCommand.map, phase = %s', self.phase)
        r_count = 0
        for record in records:
            r_count = r_count + 1
            yield record
        self.logger.info('SapAbapSummarizeCommand.map done: %d', r_count)

    @Configuration(run_in_preview=False)
    def reduce(self, records):
        self.logger.info('SapAbapSummarizeCommand.reduce, phase = %s', self.phase)
        self.logger.info('SapAbapSummarizeCommand.reduce, metadata = %s', self.metadata)
        r_count = 0

        for record in records:
            r_count = r_count + 1
            # (other processing)

        self.logger.info('SapAbapSummarizeCommand.reduce done: %d', r_count)

commands.conf:

[sapabapsummarize]
filename = sap_abap_summarize.py
chunked = true

How do I get reduce to know that all the results are in? The 'Preview' flag in the metadata is always True.

jkat54
SplunkTrust
SplunkTrust

Put this in commands.conf

[yourcommand]
run_in_preview = false

Which will disable preview

0 Karma

wegscd
Contributor

That is the SCP V1 version of SCP V2's @Configuration(run_in_preview=False), which is already been done.

The extra runs I am seeing are during preview.

0 Karma

jkat54
SplunkTrust
SplunkTrust

You're clearing the configuration prior to map though... is that the cause?

0 Karma

wegscd
Contributor

not sure, but I need to clear it out. If I put run_in_preview=False in map, I get

08-23-2017 10:29:38.001 INFO  ChunkedExternProcessor - Running process: /opt/splunk/bin/python /opt/splunk/etc/apps/wegscd_command_test/bin/sap_abap_summarize.py
08-23-2017 10:29:38.104 WARN  ChunkedExternProcessor - Unsupported metadata field sent by custom search command: run_in_preview
08-23-2017 10:29:38.104 INFO  SearchParser - PARSING: sapabapsummarize phase="map" 
0 Karma

jkat54
SplunkTrust
SplunkTrust

Interesting the whole v2 is new to me. What if you do it in commands.conf anyways?

Just curious.

0 Karma

wegscd
Contributor

It seems to behave using scp v1. I am putting together a test case; I'm getting very strange results.....

0 Karma

jbjerke_splunk
Splunk Employee
Splunk Employee

Hi wegscd

I'm having the same issue. Did you get anywhere with this? Did you managed to prevent previews somehow?

j

0 Karma
Get Updates on the Splunk Community!

Routing logs with Splunk OTel Collector for Kubernetes

The Splunk Distribution of the OpenTelemetry (OTel) Collector is a product that provides a way to ingest ...

Welcome to the Splunk Community!

(view in My Videos) We're so glad you're here! The Splunk Community is place to connect, learn, give back, and ...

Tech Talk | Elevating Digital Service Excellence: The Synergy of Splunk RUM & APM

Elevating Digital Service Excellence: The Synergy of Real User Monitoring and Application Performance ...