Archive

Ingesting data from web query that returns JSON/XML response in plaintext

Path Finder

Hello All,

I am trying to ingest data from a cloud-based 3rd party tool that returns JSON/XML in response to a web query..
Specific example as follows:
1. Enter the following URL in browser: https://toolURL/web/query.axd?type=whiteboard&format=json/etc/
2. Enter credentials.
3. Get response in browser window in JSON/XML format. There is no prompt for a file download.. The response in just in the browser as plaintext.

I want to ingest this data into Splunk Enterprise.
Is there any way I can do this out of the box in Splunk?

There is no way to install any kind of forwarder on the 3rd party tool server, nor can I ask them to include any thing in their tool that will allow HTTP Event Collection on my Splunk deployment.

Only way I figured I can do this is either via a Scripted Input or Modular input.
However, I have not used either of them earlier and don't know which one will work better.

Can someone please guide in the right direction?
Also, a proper tutorial for building a modular/scripted input would be good.

or, if there is a app that does exactly this, that would be excellent.

note #1: I do not have any kind of documentation about this 3rd party tool which can tell me if it has a REST API or not.

Thanks in advance & regards..

Tags (1)
0 Karma

New Member

data on cloud could be a mess sometime. as the data is increasing, so is the burden on servers.
if you are also facing any data related recovery problem, then you should visit UAE Data Recovery

0 Karma

SplunkTrust
SplunkTrust

@anirbandasdeb I think you should be trying out REST API Modular Input.

@Damien Dallimore 🙂

____________________________________________
| makeresults | eval message= "Happy Splunking!!!"

SplunkTrust
SplunkTrust

@anirbandasdeb As an alternate can you refer to the following Blog by Stephen Luedtke Dashboard Digest Series Episode 7 which talks about using Splunk Add On Builder to configure REST API input to Splunk as an input

Following is the Splunk documentation for Add On Builder App setup and configuration: https://docs.splunk.com/Documentation/AddonBuilder/latest/UserGuide/Overview

____________________________________________
| makeresults | eval message= "Happy Splunking!!!"
0 Karma

Path Finder

So we got around this particular problem using Scripted Input, with a python script running on a CRON schedule, executing the web query and ingesting the JSON response.

This URL was not REST compliant, nor did the 3rd party tool have any such endpoints.

Nevertheless, @niketnilay & @Damien Dallimore thank you for your help. 🙂

0 Karma

Path Finder

@niketnilay this is some truly good stuff. I will study them.

Thank you!

0 Karma

New Member

UAE Data Recoverylink text

0 Karma

Path Finder

@niketnilay

Thank you I checked it out and it seems to fit my requirements.
I will try it out and let you know.

0 Karma

Path Finder

@niketnilay @Damien Dallimore

I got around to install the App on a trial Enterprise version, configured all that is needed, scheduled it.
Its running correctly as per schedule, but with the following error, and no event in the indexes...

HTTPSConnectionPool(host='ToolHostName', port=443): Max retries exceeded with url: /web/query.axd?line=1&type=whiteboard&NegativeScrap=1&units=3&split=job&machines=E0CFAE3D-74EF-0579-8C90-E3D00F56AC70&format=json&start=20180528T060000.000 (Caused by : [Errno 11004] getaddrinfo failed)

I tried using cURL on the same URL with additional arguments and basic auth over HTTPS, and its giving the proper output.

Now I am doubting that this URL itself might not be a API URL that the REST Input App needs..

What are your views?

Regards,
Anirban.

0 Karma

Ultra Champion

If you google "errno 11004 getaddrinfo failed" you will see that you have hostname resolution errors.

0 Karma

Path Finder

@Damien Dallimore yes, I did that and double checked the settings in the app, with the same error every time.

I am also using this exact same URL+arguments with cURL [basic auth over https] and that is responding fine.

Any methods on confirming if a given URL is actually an API which is REST compliant?
Also, how does the REST API Modular Input App behave if an URL is not an API URL?

The tool itself has very sketchy documentation and the company does not say much about its workings. But I will also try to get information about this.

0 Karma

Ultra Champion

errno 11004 getaddrinfo failed : you have a DNS error.

This is at the operating system level.

The hostname can not be resolved.

perhaps you are misconfiguring your rest stanza.

please post your full rest stanza for the community to assist in troubleshooting.

0 Karma

Path Finder

what exactly is the rest stanza?

0 Karma

Ultra Champion

when you setup your rest data input , it gets saved to an inputs.conf file in a [rest] stanza. Search for it under SPLUNK_HOME/etc/*

what does it look like ?

if we have some information to look at , we may be able to help you resolve your operating systems dns lookup failures.

such as , perhaps you entered your hostname in your URL incorrectly ?

0 Karma

Path Finder

okay. let me get a hold of that.
I will get back to you on this @Damien Dallimore

0 Karma

Ultra Champion

Yes , that is exactly what you can use the REST API Modular Input for.