Splunk Search

How can I use reduce to avoid previewing a report from a slow search until it's done?

wegscd
Contributor

I have a ReportingCommand written in Python, and the SPL that feeds it is slowish.

To minimize visual churn on the screen, I don't want my command to provide intermediate results: I just want the reduce() method to get run once when all the data is ready to process.

I am using SCP 2, and I have my command annotated with run_in_preview=False, but reduce() is still getting run multiple times (with more and more input records, and preview=True is in self.metadata).

The SPL:

sourcetype=whr:sap:abap:audit | stats count, values(sourcetype) as sourcetype by sid | append [ | inputlookup whirlpool_sap_instances ] | sapabapsummarize

The command

@Configuration(requires_preop=True, run_in_preview=False)
class SapAbapSummarizeCommand(ReportingCommand):

    @Configuration()
    def map(self, records):
        self.logger.info('SapAbapSummarizeCommand.map, phase = %s', self.phase)
        r_count = 0
        for record in records:
            r_count = r_count + 1
            yield record
        self.logger.info('SapAbapSummarizeCommand.map done: %d', r_count)

    @Configuration(run_in_preview=False)
    def reduce(self, records):
        self.logger.info('SapAbapSummarizeCommand.reduce, phase = %s', self.phase)
        self.logger.info('SapAbapSummarizeCommand.reduce, metadata = %s', self.metadata)
        r_count = 0

        for record in records:
            r_count = r_count + 1
            # (other processing)

        self.logger.info('SapAbapSummarizeCommand.reduce done: %d', r_count)

commands.conf:

[sapabapsummarize]
filename = sap_abap_summarize.py
chunked = true

How do I get reduce to know that all the results are in? The 'Preview' flag in the metadata is always True.

jkat54
SplunkTrust
SplunkTrust

Put this in commands.conf

[yourcommand]
run_in_preview = false

Which will disable preview

0 Karma

wegscd
Contributor

That is the SCP V1 version of SCP V2's @Configuration(run_in_preview=False), which is already been done.

The extra runs I am seeing are during preview.

0 Karma

jkat54
SplunkTrust
SplunkTrust

You're clearing the configuration prior to map though... is that the cause?

0 Karma

wegscd
Contributor

not sure, but I need to clear it out. If I put run_in_preview=False in map, I get

08-23-2017 10:29:38.001 INFO  ChunkedExternProcessor - Running process: /opt/splunk/bin/python /opt/splunk/etc/apps/wegscd_command_test/bin/sap_abap_summarize.py
08-23-2017 10:29:38.104 WARN  ChunkedExternProcessor - Unsupported metadata field sent by custom search command: run_in_preview
08-23-2017 10:29:38.104 INFO  SearchParser - PARSING: sapabapsummarize phase="map" 
0 Karma

jkat54
SplunkTrust
SplunkTrust

Interesting the whole v2 is new to me. What if you do it in commands.conf anyways?

Just curious.

0 Karma

wegscd
Contributor

It seems to behave using scp v1. I am putting together a test case; I'm getting very strange results.....

0 Karma

jbjerke_splunk
Splunk Employee
Splunk Employee

Hi wegscd

I'm having the same issue. Did you get anywhere with this? Did you managed to prevent previews somehow?

j

0 Karma
Get Updates on the Splunk Community!

Detecting Remote Code Executions With the Splunk Threat Research Team

WATCH NOWRemote code execution (RCE) vulnerabilities pose a significant risk to organizations. If exploited, ...

Enter the Splunk Community Dashboard Challenge for Your Chance to Win!

The Splunk Community Dashboard Challenge is underway! This is your chance to showcase your skills in creating ...

.conf24 | Session Scheduler is Live!!

.conf24 is happening June 11 - 14 in Las Vegas, and we are thrilled to announce that the conference catalog ...