topic Re: How to retrieve JSON formatted data from a web server with authentication? in Getting Data In

How to retrieve JSON formatted data from a web server with authentication?

afabijan — Wed, 05 Nov 2014 09:39:59 GMT

Hi there,
i have a Restful API that returns data in JSON format. I would like to retrieve this data into Splunk.
The problem is that when I open the URL in a browser, i need to first Log IN and then i see the data in JSON.

Does anyone have an idea how to retrieve this into Splunk? Is there some way to incorporate HTTP authentication?
I used the following resources up to now:
https://apps.splunk.com/app/1311/
http://answers.splunk.com/answers/60608/injest-json-document-returned-from-an-monitoring-endpoint.html
http://answers.splunk.com/answers/118431/whats-the-best-way-to-import-custom-json-data-from-a-web-api.html

Kind regards, A

Re: How to retrieve JSON formatted data from a web server with authentication?

Damien_Dallimor — Wed, 05 Nov 2014 09:45:17 GMT

The REST API Modular Input supports numerous HTTP authentication mechanisms.

Re: How to retrieve JSON formatted data from a web server with authentication?

afabijan — Wed, 05 Nov 2014 09:48:06 GMT

Thank you. I will give it a try!!!

Re: How to retrieve JSON formatted data from a web server with authentication?

afabijan — Wed, 05 Nov 2014 14:18:20 GMT

EDIT: Seems a port issue, wait with reading this until i open a port 80 between splunk and the web server 🙂

Could you give me a bit help on using this - the URl that i get the JSON back is (in the browser) the following:
http://10.X.X.X/arapi/X.X.X.X/HPD:Help%20Desk/'1000000161'=%22INC000000010348%22?port=50000

How do i correctly enter this in the module that you developed? I see in the logs that if i enter this link, i get an exception:

HTTPConnectionPool(host='10.X.X.X', port=80): Max retries exceeded with url: /arapi/X.X.X.X-03/HPD:Help%20Desk/'1000000161'=%22INC000000010348%22?port=50000 (Caused by : [Errno 10060] A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond)

Re: How to retrieve JSON formatted data from a web server with authentication?

Damien_Dallimor — Wed, 05 Nov 2014 21:35:33 GMT

You do realise that's an encoded URL ?

It would help if you could be more accurate with the information in your question..

What is the URL of your REST Endpoint ?

What is the HTTP method for this endpoint?

What are the expected URL arguments or POST parameters ?

What is the HTTP authentication mechanism ?

How are the authentication credentials expected to be encoded in the HTTP request ?