All Apps and Add-ons

REST API Issue - Exception performing request

Builder

I am trying to pull data from Azure using REST API. Have installed REST API App https://apps.splunk.com/app/1546/ on the top of Splunk Enterprise.

After giving necessary credentials, I am getting following error in Splunkd :-

12-24-2014 04:11:32.173 +0000 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/rest_ta/bin/rest.py" Exception performing request:

12-24-2014 04:11:22.157 +0000 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/rest_ta/bin/rest.py" Exception performing request:

12-24-2014 04:11:12.141 +0000 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/rest_ta/bin/rest.py" Exception performing request:

12-24-2014 04:11:02.125 +0000 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/rest_ta/bin/rest.py" Exception performing request:

Could someone please advise me what could be the problem?

Thanks in advance.

0 Karma

Ultra Champion

As per my request , can you provide a link "detailing the actual specifications for the Azure REST endpoint you are trying to connect to"

https://msdn.microsoft.com/en-us/library/dd179355.aspx is an overview page. I have absolutely no idea what REST endpoint you are trying to hit.

Guessing , is it one of these specs ? https://msdn.microsoft.com/en-us/library/azure/ee460790.aspx

Based on the endpoint URL you have entered above in inputs.conf , maybe this ? https://msdn.microsoft.com/en-us/library/azure/ee460802.aspx

If so , then you need to authenticate by sending a management certificate in the REST requests : https://msdn.microsoft.com/en-us/library/azure/ee460782.aspx#bk_cert

That link details how to create one. You'll then need to create a custom authentication handler in the REST Mod Input to read your certificate from the file system and add it to the REST request.

An example of a custom auth handler that you can add to rest_ta/bin/authhandlers.py is :

class MyAzureCertAuthHAndler(AuthBase):
    def __init__(self,**args):
        self.cert = args['certPath']
        pass

    def __call__(self, r):
        r.cert = self.cert
        return r

Then your setup might look like below. I don't know why you were using the http proxy setting , I've stripped that out.I also added HTTP Headers because it said so in the documentation I read.

alt text

0 Karma

Builder

I added management certification into Azure console & copied that on my Splunk instance. Did below changes in authhandlers.py

class newazurecert(AuthBase):
def init(self,**args):
# setup any auth-related data here
self.cert = args['certPath']
#self.password = args['password']
pass

Now I am getting 403 error.

1-22-2015 19:45:34.488 +0530 ERROR ExecProcessor - message from "python /Users/sdange/Documents/SplunkDemo/ScriptSplunk6.2/splunk/etc/apps/rest_ta/bin/rest.py" HTTP Request error: 403 Client Error: Forbidden

1/22/15

7:45:22.196 PM

01-22-2015 19:45:22.196 +0530 ERROR ExecProcessor - message from "python /Users/sdange/Documents/SplunkDemo/ScriptSplunk6.2/splunk/etc/apps/rest_ta/bin/rest.py" HTTP Request error: 403 Client Error: Forbidden

http://azure.microsoft.com/en-in/documentation/articles/storage-monitoring-diagnosing-troubleshootin... says that I have to reissue storage token key from which I am trying to pull data but I have not given these keys in REST API.

what could be the problem?

Thanks

0 Karma

Ultra Champion

1) were my assumptions in my answer correct ? you didn't concur

2) your auth handler looks nothing like the example I provided you. Furthermore without also seeing the inputs.conf stanza , I can't be of much use.

0 Karma

Builder

My mistake. Apologize for that. You are right. I referred the same link to enter endpoint URL.

Here is my authhandler.py configuration -

class newazurecert(AuthBase):
def init(self,**args):
# setup any auth-related data here
self.cert = args['certPath']
#self.password = args['password']
pass

inputs.conf

[rest://Azure]
authtype = custom
custom
authhandler = newazurecert
custom
authhandlerargs = certPath=/Users/sdange/Documents/Apps & TA/Azure/newazurecert.cer
endpoint = https://management.core.windows.net/XXXXX/services/storageservices/satishsplcs1
httpheaderpropertys = x-ms-version=2014-10-01,Content-Type=application/xml
httpmethod = GET
index = azure
index
errorresponsecodes = 0
responsetype = text
sourcetype = csv
streaming
request = 0

0 Karma

Ultra Champion

Copy/Paste from my previous reply :

your auth handler looks nothing like the example I provided you

Repasting :

class MyAzureCertAuthHAndler(AuthBase):
     def __init__(self,**args):
         self.cert = args['certPath']
         pass

     def __call__(self, r):
         r.cert = self.cert
         return r

Furthermore , I don't know why you are specifying "sourcetype = csv". If you actually read the Azure docs you'll see that the response is XML.

0 Karma

Builder

Here is authhandler.py configuration -

class newazurecert(AuthBase):
def init(self,**args):
# setup any auth-related data here
self.cert = args['certPath']
#self.password = args['password']
pass

def __call__(self, r):
    r.cert = self.cert
    return r

inputs.conf

[rest://Azure]
authtype = custom
custom
authhandler = newazurecert
custom
authhandlerargs = certPath=/Users/sdange/Documents/Apps & TA/Azure/newazurecert.cer
endpoint = https://management.core.windows.net/XXXX/services/storageservices/satishsplcs1
httpheaderpropertys = x-ms-version=2014-10-01,Content-Type=application/xml
httpmethod = GET
index = azure
index
errorresponsecodes = 0
responsetype = text
sourcetype = xml
streaming
request = 0

0 Karma

Ultra Champion

I'd remove the spaces in the cert path too.

0 Karma

Builder

sorry, I did not understand Damien.

0 Karma

Ultra Champion

certPath=/Users/sdange/Documents/Apps & TA/Azure/newazurecert.cer

You have spaces in the cert path. I'd remove these.

Beyond that , everything else in the config looks fine.

If you are still getting HTTP 403 responses , I'd refer to the Azure documentation at that point as to why your authentication for the request could be getting declined.

0 Karma

Builder

well, that too did not work.

Now I am referring https://msdn.microsoft.com/en-us/library/hh225339.aspx to get some pointers.

0 Karma

Builder

Can someone please advise me how to fix this problem?

0 Karma

Splunk Employee
Splunk Employee

any inputs on this question?

0 Karma

Ultra Champion

You are going to have to provide more concise information for us to help you debug.

1) Splunk version
2) REST Mod Input version
3) your inputs.conf stanza you setup (sensitive info masked out) . When you enter data via the REST Mod Input set page , it gets persisted to inputs.conf. Search under etc/apps for it.
4) any more descriptive errors , search in Splunk Web : "index=_internal ExecProcessor error rest.py"

0 Karma

Ultra Champion

Furthermore , in addition to "I am trying to pull data from Azure using REST API" , can you please provide a link to the Azure documentation detailing the actual specifications for the Azure REST endpoint you are trying to connect to.

0 Karma

Builder

Hi Damien -

Here is information

1) Splunk Version - 6.2.
2) REST Mod input version - 1.3.5 https://apps.splunk.com/app/1546/
3) your inputs.conf stanza you setup (sensitive info masked out) -
[rest://Azure]
authtype = oauth2
endpoint = https://management.core.windows.net/XXXXX/services/storageservices/satishsplcs1
host =
http
method = GET
index = azure
indexerrorresponsecodes = 0
oauth1
accesstoken = XXXXXXX
response
type = text
sourcetype = csv
streamingrequest = 0
oauth1
accesstokensecret = XXXXX
httpproxy = https://manage.windowsazure.com/@XXXXXX.onmicrosoft.com#Workspaces/All/dashboard
oauth2
accesstoken = XXXXXXXX
oauth2
client_secret = XXXXXXX
disabled = 0

4) any more descriptive errors , search in Splunk Web : "index=internal ExecProcessor error rest.py”
01-21-2015 15:46:03.545 +0000 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/rest
ta/bin/rest.py" Exception performing request:
host = satishsplindx source = /opt/splunk/var/log/splunk/splunkd.log sourcetype = splunkd
1/21/15
3:45:53.533 PM
01-21-2015 15:45:53.533 +0000 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/rest_ta/bin/rest.py" Exception performing request:
host = satishsplindx source = /opt/splunk/var/log/splunk/splunkd.log sourcetype = splunkd

5) Azure link - https://msdn.microsoft.com/en-us/library/dd179355.aspx

0 Karma