custom aggregation function for stats command

[welcomerrr]

Hello,

I am building a splunk app , where I want to have my own custom aggregate function for stats command. Below is my use case let say.

| makeresults count=10
| eval event_count=random()%10
| stats mysum("event_count") as total_count

Does anyone knows how my python code should look like if its feasible to create mysum function.

Thanks!

[tscroggins]

Hi @welcomerrr,

You can often get creative with alternative SPL without restoring to custom commands. See https://community.splunk.com/t5/Splunk-Search/Product-of-a-Column/m-p/707749 for a recent example of calculating a product over a list.

Depending on the algorithm your hypothetical function implements, a solution may also be possible using a combination of an existing aggregation function and a nested eval.

[VatsalJagani]

@welcomerrr- As described by @PickleRick , you cannot create a function for stats command, but you can create the whole new custom command which might be implementing the functionality in Python.

But most of the requirements that you might have should be able to fulfilled with existing stats command function. Kindly please describe exact use-case and community should be able to help you write query without writing custom command or function.

[isoutamo]

Hi

it’s just like others already told. Maybe there is another way to fulfill your need, but this needs that we understand your real needs. 

If this cannot do otherwise then you could try to present your needs in ideas.splunk.com and hope that others give enough points to it. Then splunk could consider to create and deploy it in some future versions. I have heard that currently 70-80% of new features comes that way.

r. Ismo

[welcomerrr]

Thank you folks !! it helps.

Here is what I am trying to acheive, I want to use https://datasketches.apache.org/ Data Sketches to deserailize the skecth written into splunk. While I was able to deserailize the sketch itself but we need to merge sketches. For example I would like to merge the skecthes based on something like 

Selected fields | stats sum(total_clicks), mergeHll(unique_visitor_sketch) as merged_unique_visitors group by country

My core problem is how I could define mergeHll(unique_visitor_sketch) as in command. 

import sys
import base64

from splunklib.searchcommands import dispatch, StreamingCommand, Configuration, Option
from datasketches import hll_sketch

@Configuration()
class CreateHLLCommand(StreamingCommand):

    field = Option(require=True)

    def stream(self, records):
        """Process the streaming records and get estimate from sketch."""
        for record in records:
            # Deserialize the HLL sketch
            sketch_bytes = base64.b64decode(record[self.field])
            hll = hll_sketch.deserialize(sketch_bytes)

            record['hll_estimate'] = hll.get_estimate()
            yield record


# Dispatch the command
dispatch(CreateHLLCommand, sys.argv, sys.stdin, sys.stdout)

My custom command to deserialize the sketch. 

[isoutamo]

Just reads what HLLs are, so I’m not familiar with those and don’t really understand the whole math behind those, but I believe that I understand enough?

I think that you must replace the whole stats command with your custom command and do those calculations on python and just return answer with approximate boundaries.

Maybe there is already some libraries or examples, how you could create your own stats replacement for count and a mergeHLL part to it?

[PickleRick]

You can't create a custom aggregation function for stats. You can create your own command though.

https://dev.splunk.com/enterprise/docs/devtools/customsearchcommands/