Thank you folks !! it helps. Here is what I am trying to acheive, I want to use https://datasketches.apache.org/ Data Sketches to deserailize the skecth written into splunk. While I was able to deserailize the sketch itself but we need to merge sketches. For example I would like to merge the skecthes based on something like Selected fields | stats sum(total_clicks), mergeHll(unique_visitor_sketch) as merged_unique_visitors group by country My core problem is how I could define mergeHll(unique_visitor_sketch) as in command. import sys
import base64
from splunklib.searchcommands import dispatch, StreamingCommand, Configuration, Option
from datasketches import hll_sketch
@Configuration()
class CreateHLLCommand(StreamingCommand):
field = Option(require=True)
def stream(self, records):
"""Process the streaming records and get estimate from sketch."""
for record in records:
# Deserialize the HLL sketch
sketch_bytes = base64.b64decode(record[self.field])
hll = hll_sketch.deserialize(sketch_bytes)
record['hll_estimate'] = hll.get_estimate()
yield record
# Dispatch the command
dispatch(CreateHLLCommand, sys.argv, sys.stdin, sys.stdout) My custom command to deserialize the sketch.
... View more
