Developing for Splunk Enterprise
Highlighted

How to pass credentials within custom search command using Python SDK 1.5

Explorer

I'm building a custom search command (in Python with the SDK 1.5). From within the script (search command), I need to connect to the Splunk instance to execute some extra searches and retrieve some kv collections.

To have a service object to Splunk I use:

service = client.connect( username=, password=, app=)

But I do not want to have the username and password part of this script. I have tried "passauth" from the commands.conf, but I do not seem to get a token or session.

Is there a way to pass credentials to this type of script?

0 Karma
Highlighted

Re: How to pass credentials within custom search command using Python SDK 1.5

Contributor

Hi, If the script is being called from splunk the best way to do this is with tokens. Can you explain how you tried to use passauth? Provide some code?

0 Karma
Highlighted

Re: How to pass credentials within custom search command using Python SDK 1.5

Explorer

He tx for reply.

My code is the following:
commands.conf:
[ucmstest]
filename = ucmstest.py
supportsgetinfo = true
supports
rawargs = true
passauth = true
enableheader = true
outputheader = true
requiressrinfo = true
support
multivalues = true

ucmstest.py:

from splunklib.searchcommands import dispatch, GeneratingCommand, Configuration, Option, validators 
import sys, os
import time
import splunklib.client as client
import splunklib.results as results
import splunklib.binding as binding
import logging, logging.handlers
import splunk

###passauth=True; enableheader=True
@Configuration()
class ucmstest(GeneratingCommand):

    count = Option(require=True, validate=validators.Integer(0))
    text = Option(require=True)

    def generate(self):
        logger = logging.getLogger('UCMS dq Logging')  
        SPLUNK_HOME = os.environ['SPLUNK_HOME']
        LOGGING_DEFAULT_CONFIG_FILE = os.path.join(SPLUNK_HOME, 'etc', 'log.cfg')
        LOGGING_LOCAL_CONFIG_FILE = os.path.join(SPLUNK_HOME, 'etc', 'log-local.cfg')
        LOGGING_STANZA_NAME = 'pythondebug'
        LOGGING_FILE_NAME = "ucmsdqreconsilestats.log"
        BASE_LOG_PATH = os.path.join('var', 'log', 'splunk')
        LOGGING_FORMAT = "%(asctime)s %(levelname)-s\t%(module)s:%(lineno)d - %(message)s"
        splunk_log_handler = logging.handlers.RotatingFileHandler(os.path.join(SPLUNK_HOME, BASE_LOG_PATH, LOGGING_FILE_NAME), mode='a') 
        splunk_log_handler.setFormatter(logging.Formatter(LOGGING_FORMAT))
        logger.addHandler(splunk_log_handler)
        logmode = logging.DEBUG
        logger.setLevel(logmode)
        splunk.setupSplunkLogger(logger, LOGGING_DEFAULT_CONFIG_FILE, LOGGING_LOCAL_CONFIG_FILE, LOGGING_STANZA_NAME)
        yield {'_serial': 1, '_time': time.time(), '_raw':"this is a test"}

        logger.info("Start of logging...................................................")
        for line in sys.argv:
            logger.debug("value of argv is: %s", line)

dispatch(ucmstest, sys.argv, sys.stdin, sys.stdout, __name__)

When called from the splunk search line: |ucmstest count=5 text="hallo" it is indeed outputting what i have put in yield.
In the "ucmsdqreconsilestats.log" i can then see the values of sys.argv but no token or session is there. Also nothin in sys.stdin.
And from out this command i want to do some extra searches, retrieving a CIM model and the data of a CIM model. So i need to connect to the service by:
service = client.connect(something)
And that something is either "username=, password=" or something with token or session.

But how to get in the token or session value?

0 Karma
Highlighted

Re: How to pass credentials within custom search command using Python SDK 1.5

Contributor

Hi,
session key is passed along when script gets triggered. You can capture it by doing something like:

tokn = sys.stdin.readline().strip()

token will now hold the session key. It is urlencoded so will need to decode it. Typically ... if you are on the recent versions of splunk, something like below should work for you to connect:

tokn = tokn.replace("sessionKey=", "")
tokn = urllib2.unquote(tokn).decode('utf8')
service = client.connect(token = tokn, app = yourapp)

0 Karma
Highlighted

Re: How to pass credentials within custom search command using Python SDK 1.5

Explorer

Tx again for the comment. I'm running Splunk 6.3.2 with python SDK 1.5.0.2 just to be sure. I tried to get the information from the sys.stdin but nothing is there, it is not containing something like a session or token. So the variable "tokn" stays empty. And i do fire of the search command from out of splunk with logged in as admin.
But indeed this is how i have understood how it should work.
Once i do have the session or token i indeed can connect.

0 Karma
Highlighted

Re: How to pass credentials within custom search command using Python SDK 1.5

Contributor

ahh.. I did quick test and looks like for search commands stdin only holds the location of csv file with the requested headers. I was able to get the token by reading the csv file..

csvf = sys.stdin.readline().strip()
csvf = csvf.replace("infoPath:", "")
with open(csvf) as hfile:
        reader = csv.DictReader(hfile)
        for row in reader:
                tkn = row['_auth_token']

As mentioned above if the sessionkey might be urlencoded in which case you will need to decode before passing it to client connect. Hope this works out ..

0 Karma
Highlighted

Re: How to pass credentials within custom search command using Python SDK 1.5

Explorer

Ok this all doesn't seems to work at my environment. I have to say I'm also running El Capitan 10.11.3.

Commands.conf:

[ucmstest]
filename = ucmstest.py
supports_getinfo = true
supports_rawargs = false
passauth = true
enableheader = true
requires_srinfo = true

And ucmstest.py is:

from splunklib.searchcommands import dispatch, GeneratingCommand, Configuration, Option, validators 
import sys, os
import time
import logging, logging.handlers
import splunk
import urllib2


###passauth=True; enableheader=True
@Configuration()
class ucmstest(GeneratingCommand):

    count = Option(require=True, validate=validators.Integer(0))
    text = Option(require=True)

    def generate(self):
        logger = logging.getLogger('UCMS dq Logging')  
        SPLUNK_HOME = os.environ['SPLUNK_HOME']
        LOGGING_DEFAULT_CONFIG_FILE = os.path.join(SPLUNK_HOME, 'etc', 'log.cfg')
        LOGGING_LOCAL_CONFIG_FILE = os.path.join(SPLUNK_HOME, 'etc', 'log-local.cfg')
        LOGGING_STANZA_NAME = 'pythondebug'
        LOGGING_FILE_NAME = "ucmsdqreconsilestats.log"
        BASE_LOG_PATH = os.path.join('var', 'log', 'splunk')
        LOGGING_FORMAT = "%(asctime)s %(levelname)-s\t%(module)s:%(lineno)d - %(message)s"
        splunk_log_handler = logging.handlers.RotatingFileHandler(os.path.join(SPLUNK_HOME, BASE_LOG_PATH, LOGGING_FILE_NAME), mode='a') 
        splunk_log_handler.setFormatter(logging.Formatter(LOGGING_FORMAT))
        logger.addHandler(splunk_log_handler)
        logmode = logging.DEBUG
        logger.setLevel(logmode)
        splunk.setupSplunkLogger(logger, LOGGING_DEFAULT_CONFIG_FILE, LOGGING_LOCAL_CONFIG_FILE, LOGGING_STANZA_NAME)

        logger.info("Start of logging...................................................")
        yield {'_serial': 1, '_time': time.time(), '_raw':"this is a test"}
        logger.debug("sys.stdin is: %s",str(sys.stdin))

        #logger.debug("read %s", str(sys.stdin.read()))
        csvf = sys.stdin.readline().strip()
        logger.debug("csvf file is: %s and length %i", csvf, len(csvf))
        #csvf = csvf.replace("infoPath:", "")
        logger.debug("csvf is now: %s", csvf)
        with open(csvf) as hfile:
            logger.debug("within open section")
            reader = csv.DictReader(hfile)
            for row in reader:
                tkn = row['_auth_token']
                logger.debug("tkn is: %s", tkn)

dispatch(ucmstest, sys.argv, sys.stdin, sys.stdout, __name__)

When i do |ucmstest count=1 text="bla" in Splunk Web, I do get:

External search command 'ucmstest' returned error code 1. Script output = "error_message=IOError at "/Applications/splunkucmsdq/splunk/etc/apps/ucmsdq/bin/ucmstest.py", line 41 : [Errno 2] No such file or directory: '' _raw,__mv__raw,_time,__mv__time,_serial,__mv__serial this is a test,,1456854995.48,,1, "

And i do get the following in my "ucmsdqreconsilestats.log":

2016-03-01 19:03:43,565 INFO    __init__:168 - Using default logging config file: /Applications/splunkucmsdq/splunk/etc/log.cfg
2016-03-01 19:03:43,566 INFO    __init__:206 - Setting logger=splunk level=INFO
2016-03-01 19:03:43,566 INFO    __init__:206 - Setting logger=splunk.appserver level=INFO
2016-03-01 19:03:43,567 INFO    __init__:206 - Setting logger=splunk.appserver.controllers level=INFO
2016-03-01 19:03:43,567 INFO    __init__:206 - Setting logger=splunk.appserver.controllers.proxy level=INFO
2016-03-01 19:03:43,567 INFO    __init__:206 - Setting logger=splunk.appserver.lib level=WARN
2016-03-01 19:03:43,567 INFO    __init__:206 - Setting logger=splunk.pdfgen level=INFO
2016-03-01 19:03:43,567 INFO    __init__:206 - Setting logger=splunk.search level=INFO
2016-03-01 19:03:43,567 INFO    ucmstest:32 - Start of logging...................................................
2016-03-01 19:03:43,567 DEBUG   ucmstest:34 - sys.stdin is: <open file '<stdin>', mode 'r' at 0x107dc10c0>
2016-03-01 19:03:43,568 DEBUG   ucmstest:38 - csvf file is:  and length 0
2016-03-01 19:03:43,568 DEBUG   ucmstest:40 - csvf is now:

Can you otherwise share your code?

0 Karma
Highlighted

Re: How to pass credentials within custom search command using Python SDK 1.5

Explorer

I also have to say that i did not installed the SDK but took the "splunklib" folder out of the SDK to put it in ../bin/. folder of my app.

0 Karma
Highlighted

Re: How to pass credentials within custom search command using Python SDK 1.5

Contributor

Sure.. I have the following in my commands.conf (located in SPLUNK_HOME/etc/apps/MYAPP/local)

[tkntest]
type = python
filename = tkntst.py
supports_getinfo = false
supports_rawargs = true
passauth = true
enableheader = true
outputheader = true
requires_srinfo = true
support_multivalues = true

tkntst.py (located in SPLUNK_HOME/etc/apps/MYAPP/bin)

import sys, os, logging, csv
from logging.handlers import TimedRotatingFileHandler

splkhome = os.environ['SPLUNK_HOME']
splvar = os.path.join(splkhome, 'var', 'log', 'splunk', 'sctest.log')

### Set Logging
log = logging.getLogger('tkntst')
log.setLevel(logging.INFO)
formatter = logging.Formatter('%(asctime)s [%(levelname)s] %(message)s')
handler = logging.handlers.TimedRotatingFileHandler(splvar,when="d",interval=1,backupCount=1)
handler.setFormatter(formatter)
log.addHandler(handler)

try:
        log.info('Search commnd initiated')
        csvf = sys.stdin.readline().strip()
        csvf = csvf.replace("infoPath:", "")
        log.info('Successfully received csvf from Splunk %s' % csvf)

        with open(csvf) as hfile:
                reader = csv.DictReader(hfile)
                for row in reader:
                        tkn = row['_auth_token']
        log.info('Successfully received token from Splunk %s' % tkn)

except Exception, e:
        log.error('Unable to get session %s' % str(e))
        sys.exit()

Ran in search (MYAPP) "|tkntest"

tail /xxx/xxx/splunk/var/log/splunk/sctest.log
2016-03-01 16:58:31,932 [INFO] Search commnd initiated
2016-03-01 16:58:31,932 [INFO] Successfully received csvf from  /xxx/xxx/splunk/var/run/splunk/dispatch/1456869511.102/eternSearchResultsInfo.csv
2016-03-01 16:58:31,933 [INFO] Successfully received token from Splunk Ps^29NN_8U3WSwXdlt897cDV8bsOwxsXfN9Tqv9ATql77OONJS6S6gGI9lq3vNSW5Szwc6ltPNjFhg9HADwwpGS72CMJFTqcktxgS^8t9mrbtYoKE1T7KN

Also since you have the splunklib... checkout this answer ... seems much simpler

0 Karma
Highlighted

Re: How to pass credentials within custom search command using Python SDK 1.5

Explorer

Thanks for your code. By looking at it I do see the difference. You are not using the Splunk Python SDK where i do use that.
But by searching through the Splunk Python SDK code myself i found what i was looking for.

I told you that i need the service object to connect to Splunk and what i found out is that this Splunk Python SDK already provides me that.
When I use the following with the generate(self) function i do not even have to know the token i immediately can use the service object.

for app in self.service.apps:
      logger.debug("apps: %s", app['name'])

The Splunk Python SDK delivers a client which wraps around a couple of the REST services. If i need to call the not already covered REST services i still need the token which i found i can easily retrieve by doing:

logger.debug("session token key is: %s", self._metadata.searchinfo.session_key)
service = client.connect( token=self._metadata.searchinfo.session_key)

This is just logging the token and use it for recreate the service object myself.

So thanks for thinking with me it brought me to look deeper within the Splunk Python SDK to find these answers. I have just written them here because this is definitely someone else is looking for either.

0 Karma