How to create a custom streaming command with python sdk?

[maciep]

Every time I want to create a custom search command, I think about how easy it's going to be. Then I end up banging my head against the wall for hours on step 1.

I have a very simple custom search command following the v2 protocol but I don't understand why it doesn't work the way I think it should. I'm sure I'm missing something with python or the sdk or both, but any help would be appreciated.

In this example, I want to check each event and if it has the word "Attribute" in it, then set the blah field to "test". Otherwise set blah to "test2"

import sys,re
from splunklib.searchcommands import dispatch, StreamingCommand, Configuration

@Configuration()
class ExtractDicom(StreamingCommand):
  def stream(self, records):
    for record in records:
      if "Attribute" in record['_raw']:
         record['blah'] = "test"
      else:
         record['blah'] = "test2"

      yield record

if __name__ == "__main__":
  dispatch(ExtractDicom, sys.argv, sys.stdin, sys.stdout, __name__)

This does work. It does exactly what I think it should do. Every event has a new blah field and its value is set appropriately.

But if I change this to instead create the field blah in one condition and the field meh in the other, then splunk punches me in the face.

import sys,re
from splunklib.searchcommands import dispatch, StreamingCommand, Configuration

@Configuration()
class ExtractDicom(StreamingCommand):
  def stream(self, records):
    for record in records:
      if "Attribute" in record['_raw']:
         record['blah'] = "test"
      else:
         record['meh'] = "test2"

      yield record

if __name__ == "__main__":
  dispatch(ExtractDicom, sys.argv, sys.stdin, sys.stdout, __name__)

Now, if the first result processed has "Attribute" in it, then any subsequent event with "Attribute" in it has the field blah, great! But if the subsequent event does not have "Attribute" field in it, no meh field is created. It doesn't exist. Not great!

If I change the base search so that the first event does not have "Attribute" in it, then all of the subsequent events that don't contain "Attribute" have a field named meh. But none of the "Attribute" events have a field name blah.

It seems to be pretty consistent that if I successfully set a field for the first event in a result set, then any subsequent events that also set that same field will have the field. But if a field doesn't get set when processing the first result, then none of the subsequent events can get that field.

Ultimately, I want to create fields based on the data that is in events with "Attribute" in them. So I don't know what those fields will be called until I process those events and therefore can't set them to some default. And so from what I'm seeing, I'll only be able to set fields based on what i see in the first event...which is essentially useless.

I really hope I'm missing something simple here and that somebody has enough experience to point out my mistakes. If not, I may try to do this w/o the sdk using the old intersplunk library (I'm sure I'd find a way to get stuck taking that approach too).

Note: my commands.conf only sets the filename and chunked=true.

Thanks!

[micahkemp]

Try this:

  import sys,re
  from splunklib.searchcommands import dispatch, StreamingCommand, Configuration

 @Configuration()
 class ExtractDicom(StreamingCommand):
   def stream(self, records):
     for record in records:
       record['blah'] = None
       record['meh'] = None
       if "Attribute" in record['_raw']:
          record['blah'] = "test"
       else:
          record['meh'] = "test2"

       yield record

 if __name__ == "__main__":
   dispatch(ExtractDicom, sys.argv, sys.stdin, sys.stdout, __name__)

I ran into something incredibly similar, and the issue was that the first result returned from your custom search command dictates which fields will be handled by Splunk.

The solution is to set all of the possible fields to a default value every time, but most importantly the first time, you return a result.

[jwillaime]

---

@micahkemp wrote:
I ran into something incredibly similar, and the issue was that the first result returned from your custom search command dictates which fields will be handled by Splunk.

---

Any idea if this is a bug or a feature?