Splunk Search

Can't we use a Custom Search Command with stats in fast or smart mode ?


Hi everyone!

I had to write a script that solves the IP address from a field to the corresponding BGP AS Number.
At this point, my script is somehow working and i can use it in the Splunk search web UI like this :

index=my_index | head 1000 | iptoas asdataorigin=Ip | stats count by ASCountry

My problem is that in order to use my custom command with the stats command, I need to run the search in verbose mode. Using the smart or fast mode will throw an error because the field my scripts need is not available. My understanding is that when using the stats with fast/smart modes, the search process will only fetch the fields that are necessary to compute the stats (like the ASCountry in this case), which means my script is unable to get feeded with the needed fields, hence the failure. If I do stats on the field that my scripts needs (the IP field) it's not failing, but it's obviously not providing the stats I need since I want to make stats on AS numbers...

While I don't mind waiting a few more seconds when using the verbose mode, this problem actually means that I'm unable to use my custom command to create dashboards, since they require smart orfast mode.

My Splunk version is 6.3.2
My Splunk SDK version is 1.5.0

My script looks like this :

#!/usr/bin/env python
import os
import sys
from splunklib.searchcommands import dispatch, StreamingCommand, Configuration, Option, validators
import dns.resolver
import redis
import re

class iptoas(StreamingCommand):
    """ %(synopsis)





    srcfield = Option(doc=''' field to get IP value from ''',default='Ip')
    asdstfield = Option(doc=''' dest field for as number ''',default='ASNumber')
    asdataorigin = Option(doc=''' dest field for ias data origin (redis-cache or dns) ''',default=False)
    descdstfield=Option(doc=''' dest field for as number ''',default="ASName")
    countrydstfield=Option(doc=''' dest field for as number ''',default="ASCountry")

    def stream(self, events):
        for event in events:
        # for readability reason i'm only keeping the structure
            yield event

dispatch(iptoas, sys.argv, sys.stdin, sys.stdout, __name__)

My default/command.conf is the following :

# [commands.conf]($SPLUNK_HOME/etc/system/README/commands.conf.spec)
# Configured for Search Command Protocol version 1 by default
# Replace the contents of this file with commands-scpv2.conf to enable Search Command Protocol version 2

filename = iptoas.py
maxinputs = 10000000
chunked = true
streaming = true
supports_multivalues = true
retainsevents = true

I tried to get inspiration from the "iplocation" command which is bundled with Splunk and that happens to work perfectly in fast|smart mode, but the command is not an external command anymore. I'm not sure what to do.

Do this means that I can't write custom functions with python that works with the fast mode ? Did I forget something in the command.conf file or somewhere else ? Or maybe is there a way to force the smart mode to fetch the fields I need ? (i mean, besides using the field in the stats command).

Any help, tip or trick would be enlightening.

Thanks for your help!

1 Solution

Splunk Employee
Splunk Employee


Does it work if you specify Ip=* in your initial search? I.e.

index=my_index Ip=* | head 1000 | iptoas asdataorigin=Ip | stats count by ASCountry

Cheers, Greg.

View solution in original post


I also had this problem. This answer directly from a Slack conversation with @micahkemp , who wouldn't let me give him the karma for it.

tl;dr: the main fix is this:

   def prepare(self):
       self.configuration.required_fields = [self.source_field]

here's the full example, for context.

from splunklib.searchcommands import dispatch, Configuration, Option
from splunklib.searchcommands import StreamingCommand

class TestRequiredFieldsCommand(StreamingCommand):
   source_field = Option(require=True)
   dest_field   = Option(require=True)

   def prepare(self):
       self.configuration.required_fields = [self.source_field]

   def stream(self, records):
       for record in records:
           record[self.dest_field] = record[self.source_field]
           yield record

dispatch(TestRequiredFieldsCommand, module_name=__name__)

Splunk Employee
Splunk Employee


Does it work if you specify Ip=* in your initial search? I.e.

index=my_index Ip=* | head 1000 | iptoas asdataorigin=Ip | stats count by ASCountry

Cheers, Greg.


While this is not exactly user friendly (I mean, i'll have to explain the trick to my colleague and they might forget to use it...), I'll still use it to create my dashboard!
Thanks for your answer!

0 Karma


It should - alternatively, use the fields command to explicitly declare what fields your search needs. In fast mode (or smart when using reporting commands), Splunk deduces back-to-front what fields should be loaded... and it can't tell what fields your command uses.

0 Karma


This is working too, thanks!

0 Karma
Get Updates on the Splunk Community!

Stay Connected: Your Guide to July and August Tech Talks, Office Hours, and Webinars!

Dive into our sizzling summer lineup for July and August Community Office Hours and Tech Talks. Scroll down to ...

Edge Processor Scaling, Energy & Manufacturing Use Cases, and More New Articles on ...

Splunk Lantern is a Splunk customer success center that provides advice from Splunk experts on valuable data ...

Get More Out of Your Security Practice With a SIEM

Get More Out of Your Security Practice With a SIEMWednesday, July 31, 2024  |  11AM PT / 2PM ETREGISTER ...