All Apps and Add-ons

ERROR message from rest.py when using a custom Authentication Handler

williamchenyp
Explorer

I'm looking to add a REST API input to pull data from a vendor that we work with. I'm having issues getting the authentication part working in the rest_ta app, and am looking for help debugging this.

This vendor's API requires me to connect twice. The first time to login to obtain an authentication token. Then use that token in the second connection to run the actual API command. Here are the 2 connections if I use curl:

curl -k -v POST "https://xyzcompany.example/rest/v1/login" -H "accept: application/json" -H "Content-Type: application/json" -d "{  'clientId': '1234', 'userName': 'myUserName', 'password': 'myPassw0rd', 'isEncryptedPassword': 'false'}"

In the returned data, look for the "Authorization: Bearer ..." string

> POST /rest/v1/login HTTP/1.1
> Host: xyzcompany.example
> User-Agent: curl/7.58.0
> accept: application/json
> Content-Type: application/json
> Content-Length: 116
>
* upload completely sent off: 116 out of 116 bytes
< HTTP/1.1 200 OK
< Server: Apache-Coyote/1.1
< Authorization: Bearer aLongRandomStringOfText
< Content-Length: 0
< Date: Mon, 15 Apr 2019 13:57:01 GMT
<
* Connection #1 to host xyzcompany.example left intact

Use the Authorization: Bearer string to run, for example, a GET /events command:

curl -X GET "https://xyzcompany.example/rest/v1/events/" -H "accept: application/json" -H  "Content-Type: application/json" -k --header "Authorization:  Bearer aLongRandomStringOfText"

This should return a list of events in JSON format that I want to index in to Splunk. I'm able to use the rest_ta app to create a new data input stanza as follows:

[rest://XYZ GET All Events NoAuth]
activation_key = myActivationKey
auth_type = none
endpoint = https://xyzcompany.example/rest/v1/events/
http_header_propertys = Authorization=Bearer aLongRandomStringOfText
http_method = GET
index_error_response_codes = 1
response_handler =
response_type = json
sequential_mode = 0
sourcetype = ntt:event
streaming_request = 0
cookies = JSESSIONID=D07F70A3AC3821C15EE462DFFEB91009.10.250.74.10
disabled = 0

This only works if I manually generate the Authentication Bearer key and paste it to the HTTP Header Properties field. The problem is the key expires after 8 hours, so I have to manually update the key to get new data.

In order to have Splunk automate this, I understand I need to create a custom Authentication Type in the authhandlers.py file. I found some sample code from Answers from users doing similar type of API calls, but the custom handler errors out for me. I'm a bit stuck at this point and am hoping to find help to debug this for me.

To use the custom Authentication Handler, I modify the input stanza to add the "XYZAuthHandler" custom type. Then in the Custom Authentication Handler Arguments, I typed in "password=myPassw0rd" (without the quotes). This is so that when I change the password, I don't need to modify the authhandler.py file. Here is the snippet of code that's in my authhandlers.py file.

from requests.auth import AuthBase
import hmac
import base64
import hashlib
import urlparse
import urllib
import requests
import json

#add your custom auth handler class to this module

class XYZAuthHandler(AuthBase):
    def __init__(self,**args):
        self.password = args['password']
        pass

    def __call__(self, r):
        url = 'https://xyzcompany.example/rest/v1/login'
        data = {
                'clientID':1234,
                'userName':'myUserName',
                'password':self.password,
                'isEncryptedPassword':'false',
                }
        req_args = {"verify" : False}
        headers={'Content-Type':'application/json'}
        auth_response = requests.post(url,data=json.dumps(data),headers=headers,**req_args)
        response_json = json.loads(auth_response.text)
        token_key = response_json["Authorization"]
        r.headers['Authorization'] = token_key
        return r

class MyEncryptedCredentialsAuthHAndler(AuthBase):
[snip]

I've verified that there are no mixture of spaces and tabs to indent the code. All the indents are space characters.

The error I'm getting from _internal index is:

04-25-2019 22:38:01.712 -0400 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/rest_ta/bin/rest.py" Exception performing request: 'Authorization'

"Authorization" is suppose to be the key to parse out the authentication token, but it looks like it's being treated like an "action" of some kind. I'm not sure how to fix this....

This is really my first time working in Python, so I hope the code I found is actually working for what I need to do. They looked OK from me Googling for those specific Python syntax, which is about as far as I can take from what I know of Python.

Any help debugging this would be greatly appreciated.

Thanks!

0 Karma
1 Solution

williamchenyp
Explorer

Qapla'! Problem solved!

Thanks for the tip from a comment from @David.rose in https://answers.splunk.com/answers/303845/is-it-possible-to-obtain-session-keys-to-access-a.html.

I took the snippet of code from the XYZAuthHandler class, and used it in a test.py file to see what the POST returns. In my case, the "Authorization" string that I'm looking for is in the headers part of the response. So I need to use auth_response.headers instead of auth_response.text. auth_response.text, doesn't have anything in there, and that's why my original code from above errors out.

I also found out that I don't need to do a "json.loads" on the headers. I just have to pull the auth string using auth_response.headers['Authorization'].

So here's the code that I added to authhandlers.py file. Hope it will be helpful to somebody someday.

import requests
import json

#add your custom auth handler class to this module

class XYZAuthHandler(AuthBase):
    def __init__(self,**args):
        self.password = args['password']
        pass

    def __call__(self, r):
        if not 'Authorization' in r.headers:
            #perform auth
            auth_url = 'https://xyzcompany.example/rest/v1/login'
            credentials = {'clientId':'1234','userName':'myUserName','password':self.password,'isEncryptedPassword':'false'}
            req_args = {"verify" : False}
            headers = {'content-type': 'application/json'}
            auth_response = requests.post(auth_url,data=json.dumps(credentials),headers=headers,**req_args)
            # auth_response = requests.post(auth_url,data=credentials,headers=headers,**req_args)
            # response_json = json.loads(auth_response.text)
            # session_key = response_json["Authorization"]
            session_key = auth_response.headers['Authorization']
            r.headers['Authorization'] = session_key
            return r

Here's the input stanza to go with the working XYZAuthHandler. I think the "cookies" part is added by Splunk, so yours should be unique.

[rest://NTT GET All Events]
activation_key = mySecretKey
auth_type = custom
custom_auth_handler = XYZAuthHandler
custom_auth_handler_args = password=myAPICredentialPassw0rd
endpoint = https://xyzcompany.example/rest/v1/events/
host = xyzserver
http_method = GET
index_error_response_codes = 1
response_type = json
sequential_mode = 0
sourcetype = xyz:event
streaming_request = 0
http_header_propertys =
disabled = 0
response_filter_pattern =
polling_interval = 60
backoff_time = 10
request_timeout = 30
response_handler =
cookies = JSESSIONID=xxxxxx

View solution in original post

0 Karma

williamchenyp
Explorer

Qapla'! Problem solved!

Thanks for the tip from a comment from @David.rose in https://answers.splunk.com/answers/303845/is-it-possible-to-obtain-session-keys-to-access-a.html.

I took the snippet of code from the XYZAuthHandler class, and used it in a test.py file to see what the POST returns. In my case, the "Authorization" string that I'm looking for is in the headers part of the response. So I need to use auth_response.headers instead of auth_response.text. auth_response.text, doesn't have anything in there, and that's why my original code from above errors out.

I also found out that I don't need to do a "json.loads" on the headers. I just have to pull the auth string using auth_response.headers['Authorization'].

So here's the code that I added to authhandlers.py file. Hope it will be helpful to somebody someday.

import requests
import json

#add your custom auth handler class to this module

class XYZAuthHandler(AuthBase):
    def __init__(self,**args):
        self.password = args['password']
        pass

    def __call__(self, r):
        if not 'Authorization' in r.headers:
            #perform auth
            auth_url = 'https://xyzcompany.example/rest/v1/login'
            credentials = {'clientId':'1234','userName':'myUserName','password':self.password,'isEncryptedPassword':'false'}
            req_args = {"verify" : False}
            headers = {'content-type': 'application/json'}
            auth_response = requests.post(auth_url,data=json.dumps(credentials),headers=headers,**req_args)
            # auth_response = requests.post(auth_url,data=credentials,headers=headers,**req_args)
            # response_json = json.loads(auth_response.text)
            # session_key = response_json["Authorization"]
            session_key = auth_response.headers['Authorization']
            r.headers['Authorization'] = session_key
            return r

Here's the input stanza to go with the working XYZAuthHandler. I think the "cookies" part is added by Splunk, so yours should be unique.

[rest://NTT GET All Events]
activation_key = mySecretKey
auth_type = custom
custom_auth_handler = XYZAuthHandler
custom_auth_handler_args = password=myAPICredentialPassw0rd
endpoint = https://xyzcompany.example/rest/v1/events/
host = xyzserver
http_method = GET
index_error_response_codes = 1
response_type = json
sequential_mode = 0
sourcetype = xyz:event
streaming_request = 0
http_header_propertys =
disabled = 0
response_filter_pattern =
polling_interval = 60
backoff_time = 10
request_timeout = 30
response_handler =
cookies = JSESSIONID=xxxxxx

View solution in original post

0 Karma
Register for .conf21 Now! Go Vegas or Go Virtual!

How will you .conf21? You decide! Go in-person in Las Vegas, 10/18-10/21, or go online with .conf21 Virtual, 10/19-10/20.