topic Re: Ingesting data from web query that returns JSON/XML response in plaintext in Getting Data In

Ingesting data from web query that returns JSON/XML response in plaintext

anirbandasdeb — Sat, 09 Jun 2018 11:41:26 GMT

Hello All,

I am trying to ingest data from a cloud-based 3rd party tool that returns JSON/XML in response to a web query..
Specific example as follows:
1. Enter the following URL in browser: https://toolURL/web/query.axd?type=whiteboard&format=json/etc/
2. Enter credentials.
3. Get response in browser window in JSON/XML format. There is no prompt for a file download.. The response in just in the browser as plaintext.

I want to ingest this data into Splunk Enterprise.
Is there any way I can do this out of the box in Splunk?

There is no way to install any kind of forwarder on the 3rd party tool server, nor can I ask them to include any thing in their tool that will allow HTTP Event Collection on my Splunk deployment.

Only way I figured I can do this is either via a Scripted Input or Modular input.
However, I have not used either of them earlier and don't know which one will work better.

Can someone please guide in the right direction?
Also, a proper tutorial for building a modular/scripted input would be good.

or, if there is a app that does exactly this, that would be excellent.

note #1: I do not have any kind of documentation about this 3rd party tool which can tell me if it has a REST API or not.

Thanks in advance & regards..

Re: Ingesting data from web query that returns JSON/XML response in plaintext

niketn — Sat, 09 Jun 2018 16:59:56 GMT

@anirbandasdeb I think you should be trying out REST API Modular Input.

@Damien Dallimore 🙂

Re: Ingesting data from web query that returns JSON/XML response in plaintext

Damien_Dallimor — Sat, 09 Jun 2018 21:30:07 GMT

Yes , that is exactly what you can use the REST API Modular Input for.

Re: Ingesting data from web query that returns JSON/XML response in plaintext

anirbandasdeb — Mon, 11 Jun 2018 06:38:26 GMT

@niketnilay

Thank you I checked it out and it seems to fit my requirements.
I will try it out and let you know.

Re: Ingesting data from web query that returns JSON/XML response in plaintext

anirbandasdeb — Thu, 21 Jun 2018 07:45:50 GMT

@niketnilay @Damien Dallimore

I got around to install the App on a trial Enterprise version, configured all that is needed, scheduled it.
Its running correctly as per schedule, but with the following error, and no event in the indexes...

HTTPSConnectionPool(host='ToolHostName', port=443): Max retries exceeded with url: /web/query.axd?line=1&type=whiteboard&NegativeScrap=1&units=3&split=job&machines=E0CFAE3D-74EF-0579-8C90-E3D00F56AC70&format=json&start=20180528T060000.000 (Caused by : [Errno 11004] getaddrinfo failed)

I tried using cURL on the same URL with additional arguments and basic auth over HTTPS, and its giving the proper output.

Now I am doubting that this URL itself might not be a API URL that the REST Input App needs..

What are your views?

Regards,
Anirban.

Re: Ingesting data from web query that returns JSON/XML response in plaintext

Damien_Dallimor — Thu, 21 Jun 2018 09:58:09 GMT

If you google "errno 11004 getaddrinfo failed" you will see that you have hostname resolution errors.

Re: Ingesting data from web query that returns JSON/XML response in plaintext

anirbandasdeb — Fri, 22 Jun 2018 04:34:07 GMT

@Damien Dallimore yes, I did that and double checked the settings in the app, with the same error every time.

I am also using this exact same URL+arguments with cURL [basic auth over https] and that is responding fine.

Any methods on confirming if a given URL is actually an API which is REST compliant?
Also, how does the REST API Modular Input App behave if an URL is not an API URL?

The tool itself has very sketchy documentation and the company does not say much about its workings. But I will also try to get information about this.

Re: Ingesting data from web query that returns JSON/XML response in plaintext

Damien_Dallimor — Fri, 22 Jun 2018 04:43:00 GMT

errno 11004 getaddrinfo failed : you have a DNS error.

This is at the operating system level.

The hostname can not be resolved.

perhaps you are misconfiguring your rest stanza.

please post your full rest stanza for the community to assist in troubleshooting.

Re: Ingesting data from web query that returns JSON/XML response in plaintext

anirbandasdeb — Fri, 22 Jun 2018 05:29:50 GMT

what exactly is the rest stanza?

Re: Ingesting data from web query that returns JSON/XML response in plaintext

niketn — Fri, 22 Jun 2018 05:35:19 GMT

@anirbandasdeb As an alternate can you refer to the following Blog by Stephen Luedtke Dashboard Digest Series Episode 7 which talks about using Splunk Add On Builder to configure REST API input to Splunk as an input

Following is the Splunk documentation for Add On Builder App setup and configuration: https://docs.splunk.com/Documentation/AddonBuilder/latest/UserGuide/Overview

Re: Ingesting data from web query that returns JSON/XML response in plaintext

rossgeller99 — Fri, 22 Jun 2018 05:44:02 GMT

data on cloud could be a mess sometime. as the data is increasing, so is the burden on servers.
if you are also facing any data related recovery problem, then you should visit UAE Data Recovery

Re: Ingesting data from web query that returns JSON/XML response in plaintext

rossgeller99 — Fri, 22 Jun 2018 05:45:43 GMT

UAE Data Recoverylink text

Re: Ingesting data from web query that returns JSON/XML response in plaintext

Damien_Dallimor — Fri, 22 Jun 2018 05:50:09 GMT

when you setup your rest data input , it gets saved to an inputs.conf file in a [rest] stanza. Search for it under SPLUNK_HOME/etc/*

what does it look like ?

if we have some information to look at , we may be able to help you resolve your operating systems dns lookup failures.

such as , perhaps you entered your hostname in your URL incorrectly ?

Re: Ingesting data from web query that returns JSON/XML response in plaintext

anirbandasdeb — Fri, 22 Jun 2018 05:55:40 GMT

okay. let me get a hold of that.
I will get back to you on this @Damien Dallimore

Re: Ingesting data from web query that returns JSON/XML response in plaintext

anirbandasdeb — Fri, 22 Jun 2018 08:55:42 GMT

@niketnilay this is some truly good stuff. I will study them.

Thank you!

Re: Ingesting data from web query that returns JSON/XML response in plaintext

anirbandasdeb — Tue, 24 Jul 2018 06:01:09 GMT

So we got around this particular problem using Scripted Input, with a python script running on a CRON schedule, executing the web query and ingesting the JSON response.

This URL was not REST compliant, nor did the 3rd party tool have any such endpoints.

Nevertheless, @niketnilay & @Damien Dallimore thank you for your help. 🙂