Splunk Search

How do I maintain multivalue fields in a steaming custom command?

curtisb1024
Path Finder

I'm working on a streaming custom command that converts a field containing binary to a multivalue field of the binary bit values, but whenever an existing multivalue field is passed in to my command it seems to get converted to a space separated string.

| append [| stats count as dummy] | eval dummymv="test1 test2 test3" | makemv dummymv delim=" " | eval test1="1100101" | eval test2=test1

alt text

And after running it through my command...

| append [| stats count as dummy] | eval dummymv="test1 test2 test3" | makemv dummymv delim=" " | eval test1="1100101" | eval test2=test1 | frombitmask field=test1

alt text

My command:

def stream(self, events):
    for event in events:
        #skip the event if it doesn't contain the field, or the field isn't binary
        if self.field not in event.keys() or re.match("^(0b)?[0-1]+$", event[self.field], re.I) is None:
            yield event
            continue

        try:
            initial = event[self.field];
            event[self.field] = []
            for i, c in enumerate(reversed(initial)):
                if c == "0" or c == "1":
                    value = int(c, 2)*((2**i))
                    if value > 0:
                        event[self.field].append(str(value))
                else:
                    break
        except Exception, e:
           if not self.suppress_error:
               raise e
    yield event

commands.conf

[frombitmask]
type = python
filename = frombitmask.py
streaming = true
supports_getinfo = true
supports_multivalues = true
outputheader = true

I've tried the sample streaming command (countmatches) in the python SDK and it seems to suffer from this issue as well.

I also read through the SDK documentation on searchcommands, and while point #8 mentions fields using the naming convention "__mv_somefieldname" for multivalue fields, the events dictionary (in my command above) never contains any fields named like this.

How can I prevent multivalue fields from losing their multivaluedness in a streaming custom command?

Update

Further testing reveals that, while the __GETINFO__ call is indicating to Splunk that multivalues are supported:

stdout

   changes_colorder,clear_required_fields,enableheader,generating,local,maxinputs,needs_empty_results,outputheader,overrides_timeorder,passauth,perf_warn_limit,required_fields,requires_srinfo,retainsevents,run_in_preview,stderr_dest,streaming,supports_multivalues,supports_rawargs,__mv_changes_colorder,__mv_clear_required_fields,__mv_enableheader,__mv_generating,__mv_local,__mv_maxinputs,__mv_needs_empty_results,__mv_outputheader,__mv_overrides_timeorder,__mv_passauth,__mv_perf_warn_limit,__mv_required_fields,__mv_requires_srinfo,__mv_retainsevents,__mv_run_in_preview,__mv_stderr_dest,__mv_streaming,__mv_supports_multivalues,__mv_supports_rawargs
1,0,1,0,0,0,1,1,0,0,0,,0,1,1,log,1,1,1,,,,,,,,,,,,,,,,,,,

Splunk is not sending any __mv_ fields when passing data to the custom command:

stdin

dummy,dummymv,test1,test2
0,"test1 test2 test3",1100101,1100101

I'm running Splunk 6.2.0, is this perhaps a known issue?

1 Solution

curtisb1024
Path Finder

This appears to be a bug in 6.2.0. Upgrading to 6.2.4 fixes this issue.

View solution in original post

curtisb1024
Path Finder

This appears to be a bug in 6.2.0. Upgrading to 6.2.4 fixes this issue.

View solution in original post

Did you miss .conf21 Virtual?

Good news! The event's keynotes and many of its breakout sessions are now available online, and still totally FREE!