REST API Modular Input - 401 Client Unauthorized

[hpintelliflo]

I'm trying to get the REST Input to work with Google Nest API which has a space in one of the headers which I think is causing an issue. I can get other REST APIs to work on the same server. The header is the Authorization one which includes Bearer and then a key

From postman I can get to the Nest API from the server so it's not a network issue.

But splunkd.log is giving me

09-29-2017 15:10:52.452 +0000 ERROR ExecProcessor - message from "python "C:\Program Files\Splunk\etc\apps\rest_ta\bin\rest.py"" HTTP Request error: 401 Client Error: Unauthorized

I’ve tried putting inverted commas around it but that hasn’t fixed it. I have also tried replacing the space with %20

The inputs.conf stanza is

[rest://Nest]
auth_type = none
endpoint = https://developer-api.nest.com/
http_header_propertys = Authorization=Bearer c.hp9b{rest of key}
http_method = GET
index = nest
index_error_response_codes = 0
response_type = json
sequential_mode = 0
sourcetype = _json
streaming_request = 0
disabled = 0

[Damien_Dallimor · Accepted solution]

You need to add a custom response handler in rest_ta/bin/responsehandlers.py and declare this handler to be applied in your stanza setup.

This custom handler would process the response from request 1 , get the the url from the location header for request 2 and send off request 2.

[hpintelliflo]

Awesome. Many thanks indeed for all your help on this one!

[Damien_Dallimor]

Some of the code examples dont have the "c." part , what happens if you remove that ?

Also try adding in the Content-Type=application/json header also as per the examples in the docs

[hpintelliflo]

Guys - i think i have worked it out ... Nest does a 307 redirect. I suspect the module is not sending the headers on the redirect request. This Python is working

import httplib

headers = {"Authorization": "Bearer c.{INSERT KEY}"}

conn = httplib.HTTPSConnection("developer-api.nest.com")
conn.request("GET", "/", "", headers)
response = conn.getresponse()

url2 = response.getheader("location")
url2 = url2[8:-1]

conn2 = httplib.HTTPSConnection(url2)
conn2.request("GET", "/", "", headers)
response2 = conn2.getresponse()

print response2.read()
conn.close()
conn2.close()

Any thoughts?

[hpintelliflo]

Thanks Damien. Will try. I’ve confirmed wih postman that the content type header isn’t required.

[Damien_Dallimor]

I can't really see anything else based of that inputs.conf snippet.
Maybe ensure that there is no hidden whitespace after your token and/or put in the content type header anyway as is in the Python 2.7 examples on the docs site.

[hpintelliflo]

Guys I've tried all of these but nothing has worked ... any more ideas?

[Damien_Dallimor]

Can you compare the HTTP POST/GET data you see on the actual wire with a successful request vs a non successful request.This should tell you exactly what , if any, differences there may be at the client end ie: using ngrep, wireshark , tcpdump .. or maybe even Splunk Stream 🙂

Some examples here of using ngrep and tcpdump : https://stackoverflow.com/questions/9241391/how-to-capture-all-the-http-packets-using-tcpdump

[jkat54]

I’ll second this approach

[jkat54]

This reminds me of a BBQ temperature sensor I helped integrate with Splunk a year or two ago. In the end we found the temp sensors API wanted the Authorization token in the headers to be lowercase authorization. Not sure how to hack that into the rest_ta, but it's worth a shot in postman to see if you can replicate it.