I have a ReportingCommand written in Python, and the SPL that feeds it is slowish.
To minimize visual churn on the screen, I don't want my command to provide intermediate results: I just want the reduce() method to get run once when all the data is ready to process.
I am using SCP 2, and I have my command annotated with run_in_preview=False, but reduce() is still getting run multiple times (with more and more input records, and preview=True is in self.metadata).
The SPL:
sourcetype=whr:sap:abap:audit | stats count, values(sourcetype) as sourcetype by sid | append [ | inputlookup whirlpool_sap_instances ] | sapabapsummarize
The command
@Configuration(requires_preop=True, run_in_preview=False)
class SapAbapSummarizeCommand(ReportingCommand):
@Configuration()
def map(self, records):
self.logger.info('SapAbapSummarizeCommand.map, phase = %s', self.phase)
r_count = 0
for record in records:
r_count = r_count + 1
yield record
self.logger.info('SapAbapSummarizeCommand.map done: %d', r_count)
@Configuration(run_in_preview=False)
def reduce(self, records):
self.logger.info('SapAbapSummarizeCommand.reduce, phase = %s', self.phase)
self.logger.info('SapAbapSummarizeCommand.reduce, metadata = %s', self.metadata)
r_count = 0
for record in records:
r_count = r_count + 1
# (other processing)
self.logger.info('SapAbapSummarizeCommand.reduce done: %d', r_count)
commands.conf:
[sapabapsummarize]
filename = sap_abap_summarize.py
chunked = true
How do I get reduce to know that all the results are in? The 'Preview' flag in the metadata is always True.
Put this in commands.conf
[yourcommand]
run_in_preview = false
Which will disable preview
That is the SCP V1 version of SCP V2's @Configuration(run_in_preview=False), which is already been done.
The extra runs I am seeing are during preview.
You're clearing the configuration prior to map though... is that the cause?
not sure, but I need to clear it out. If I put run_in_preview=False in map, I get
08-23-2017 10:29:38.001 INFO ChunkedExternProcessor - Running process: /opt/splunk/bin/python /opt/splunk/etc/apps/wegscd_command_test/bin/sap_abap_summarize.py
08-23-2017 10:29:38.104 WARN ChunkedExternProcessor - Unsupported metadata field sent by custom search command: run_in_preview
08-23-2017 10:29:38.104 INFO SearchParser - PARSING: sapabapsummarize phase="map"
Interesting the whole v2 is new to me. What if you do it in commands.conf anyways?
Just curious.
It seems to behave using scp v1. I am putting together a test case; I'm getting very strange results.....
Hi wegscd
I'm having the same issue. Did you get anywhere with this? Did you managed to prevent previews somehow?
j