multireport vs. appendpipe

actionabledata — Fri, 09 Jul 2021 20:31:52 GMT

Code Architecture:

common code generating initial results; generate unique key for each grouping
multireport or appendpipe
stanza 1 with its own set of stats, evals, etc.; uses the key from the common code
stanza 2 with its own set of stats, evals, etc.; uses the key from the common code
Aggregates the resulting data from the common, stanza1, stanza2 results using the key

Issue Statement

common code + stanza 1 takes about 1 min to execute
common code + stanza 2 takes about 1 min to execute
common code + stanza 1 + stanza2 using either a multireport or appendpipe takes about 17 min

[Q] Does this huge execution time difference make sense?

I have attached a few images to show how I think multireport and appendpipe work.

[Q] Is my understanding accurate?

The pre-appendpipe SPL reads the data from the index, filters the data, creates some initial fields using streamstats and eventstats and creates a key that is unique per the overall groupings correlated within this code.

Lines 1 and 2 are identical and originate from the pre-multireport SPL. These results are presented to the stanza1 and stanza2 SPL.

Lines 3 and 4 are independent results from stanza1 and stanza2 respectively

stanza1 and stanza2 execute mutually exclusive from one another

The sort and stats clauses within stanza1 and stanza2 are quite different but the one does NOT impact the other.

The final aggregation software ties all the data together based on a common key.

The pre-appendpipe SPL reads the data from the index, filters the data, creates some initial fields using streamstats and eventstats and creates a key that is unique per the overall groupings correlated within this code.

Lines 1 and 2 and 3 are identical and originate from the pre-appendpipe SPL. These results are presented to the stanza1 and stanza2 SPL.

Lines 3 and 4 CAN be removed if I filter the input data with a where clause and the flag I called "which" associated with each set of data.

Lines 5 and 6 are independent results from stanza1 and stanza2 respectively

stanza1 and stanza2 execute mutually exclusive from one another.

The stats clauses within stanza1 and stanza2 are quite different but the one does NOT impact the other.

The final aggregation software ties all the data together based on a common key.

Re: multireport vs. appendpipe

actionabledata — Fri, 09 Jul 2021 21:24:16 GMT

Update to the appendpipe version of code

I eliminated stanza2 and the final aggregation SPL reducing the overall code to just the pre-appendpipe SPL and stanza 1 but leaving the appendpipe nomenclature in the code.

Total execution time = 486 sec

Then for this exact same search, I eliminated the appendpipe syntax. Everything is the same except for the | appendpipe and [ ] syntax.

Total execution time = 77 sec

The overhead to using appendpipe is HUGE.

I suspect the same is true for using multireport.

topic Re: multireport vs. appendpipe in Reporting

multireport vs. appendpipe

Re: multireport vs. appendpipe