Can't we use a Custom Search Command with stats in fast or smart mode ?

[simond_vr]

Hi everyone!

I had to write a script that solves the IP address from a field to the corresponding BGP AS Number.
At this point, my script is somehow working and i can use it in the Splunk search web UI like this :

index=my_index | head 1000 | iptoas asdataorigin=Ip | stats count by ASCountry

My problem is that in order to use my custom command with the stats command, I need to run the search in verbose mode. Using the smart or fast mode will throw an error because the field my scripts need is not available. My understanding is that when using the stats with fast/smart modes, the search process will only fetch the fields that are necessary to compute the stats (like the ASCountry in this case), which means my script is unable to get feeded with the needed fields, hence the failure. If I do stats on the field that my scripts needs (the IP field) it's not failing, but it's obviously not providing the stats I need since I want to make stats on AS numbers...

While I don't mind waiting a few more seconds when using the verbose mode, this problem actually means that I'm unable to use my custom command to create dashboards, since they require smart orfast mode.

My Splunk version is 6.3.2
My Splunk SDK version is 1.5.0

My script looks like this :

#!/usr/bin/env python
import os
import sys
from splunklib.searchcommands import dispatch, StreamingCommand, Configuration, Option, validators
import dns.resolver
import redis
import re

@Configuration()
class iptoas(StreamingCommand):
    """ %(synopsis)

    ##Syntax

    %(syntax)

    ##Description

    %(description)

    """
    srcfield = Option(doc=''' field to get IP value from ''',default='Ip')
    asdstfield = Option(doc=''' dest field for as number ''',default='ASNumber')
    asdataorigin = Option(doc=''' dest field for ias data origin (redis-cache or dns) ''',default=False)
    descdstfield=Option(doc=''' dest field for as number ''',default="ASName")
    countrydstfield=Option(doc=''' dest field for as number ''',default="ASCountry")


    def stream(self, events):
        for event in events:
        # for readability reason i'm only keeping the structure
            event[self.asdstfield]=do_the_magic(event[self.srcfield])
            yield event

dispatch(iptoas, sys.argv, sys.stdin, sys.stdout, __name__)

My default/command.conf is the following :

# [commands.conf]($SPLUNK_HOME/etc/system/README/commands.conf.spec)
# Configured for Search Command Protocol version 1 by default
# Replace the contents of this file with commands-scpv2.conf to enable Search Command Protocol version 2

[iptoas]
filename = iptoas.py
maxinputs = 10000000
chunked = true
streaming = true
supports_multivalues = true
retainsevents = true

I tried to get inspiration from the "iplocation" command which is bundled with Splunk and that happens to work perfectly in fast|smart mode, but the command is not an external command anymore. I'm not sure what to do.

Do this means that I can't write custom functions with python that works with the fast mode ? Did I forget something in the command.conf file or somewhere else ? Or maybe is there a way to force the smart mode to fetch the fields I need ? (i mean, besides using the field in the stats command).

Any help, tip or trick would be enlightening.

Thanks for your help!

[supersleepwalke]

I also had this problem. This answer directly from a Slack conversation with @micahkemp , who wouldn't let me give him the karma for it.

tl;dr: the main fix is this:

   def prepare(self):
       self.configuration.required_fields = [self.source_field]

here's the full example, for context.

from splunklib.searchcommands import dispatch, Configuration, Option
from splunklib.searchcommands import StreamingCommand

@Configuration(distributed=False)
class TestRequiredFieldsCommand(StreamingCommand):
   source_field = Option(require=True)
   dest_field   = Option(require=True)

   def prepare(self):
       self.configuration.required_fields = [self.source_field]

   def stream(self, records):
       for record in records:
           record[self.dest_field] = record[self.source_field]
           yield record

dispatch(TestRequiredFieldsCommand, module_name=__name__)

[gwiley_splunk · Accepted solution]

Hi,

Does it work if you specify Ip=* in your initial search? I.e.

index=my_index Ip=* | head 1000 | iptoas asdataorigin=Ip | stats count by ASCountry

Cheers, Greg.

[simond_vr]

Hi!
While this is not exactly user friendly (I mean, i'll have to explain the trick to my colleague and they might forget to use it...), I'll still use it to create my dashboard!
Thanks for your answer!

[martin_mueller]

It should - alternatively, use the fields command to explicitly declare what fields your search needs. In fast mode (or smart when using reporting commands), Splunk deduces back-to-front what fields should be loaded... and it can't tell what fields your command uses.

[simond_vr]

This is working too, thanks!