Getting Data In

Ingesting data from web query that returns JSON/XML response in plaintext

anirbandasdeb
Path Finder

Hello All,

I am trying to ingest data from a cloud-based 3rd party tool that returns JSON/XML in response to a web query..
Specific example as follows:
1. Enter the following URL in browser: https://toolURL/web/query.axd?type=whiteboard&format=json/etc/
2. Enter credentials.
3. Get response in browser window in JSON/XML format. There is no prompt for a file download.. The response in just in the browser as plaintext.

I want to ingest this data into Splunk Enterprise.
Is there any way I can do this out of the box in Splunk?

There is no way to install any kind of forwarder on the 3rd party tool server, nor can I ask them to include any thing in their tool that will allow HTTP Event Collection on my Splunk deployment.

Only way I figured I can do this is either via a Scripted Input or Modular input.
However, I have not used either of them earlier and don't know which one will work better.

Can someone please guide in the right direction?
Also, a proper tutorial for building a modular/scripted input would be good.

or, if there is a app that does exactly this, that would be excellent.

note #1: I do not have any kind of documentation about this 3rd party tool which can tell me if it has a REST API or not.

Thanks in advance & regards..

Tags (1)
0 Karma

rossgeller99
New Member

data on cloud could be a mess sometime. as the data is increasing, so is the burden on servers.
if you are also facing any data related recovery problem, then you should visit UAE Data Recovery

0 Karma

niketn
Legend

@anirbandasdeb I think you should be trying out REST API Modular Input.

@Damien Dallimore 🙂

____________________________________________
| makeresults | eval message= "Happy Splunking!!!"

niketn
Legend

@anirbandasdeb As an alternate can you refer to the following Blog by Stephen Luedtke Dashboard Digest Series Episode 7 which talks about using Splunk Add On Builder to configure REST API input to Splunk as an input

Following is the Splunk documentation for Add On Builder App setup and configuration: https://docs.splunk.com/Documentation/AddonBuilder/latest/UserGuide/Overview

____________________________________________
| makeresults | eval message= "Happy Splunking!!!"
0 Karma

anirbandasdeb
Path Finder

So we got around this particular problem using Scripted Input, with a python script running on a CRON schedule, executing the web query and ingesting the JSON response.

This URL was not REST compliant, nor did the 3rd party tool have any such endpoints.

Nevertheless, @niketnilay & @Damien Dallimore thank you for your help. 🙂

0 Karma

anirbandasdeb
Path Finder

@niketnilay this is some truly good stuff. I will study them.

Thank you!

0 Karma

rossgeller99
New Member

UAE Data Recoverylink text

0 Karma

anirbandasdeb
Path Finder

@niketnilay

Thank you I checked it out and it seems to fit my requirements.
I will try it out and let you know.

0 Karma

anirbandasdeb
Path Finder

@niketnilay @Damien Dallimore

I got around to install the App on a trial Enterprise version, configured all that is needed, scheduled it.
Its running correctly as per schedule, but with the following error, and no event in the indexes...

HTTPSConnectionPool(host='ToolHostName', port=443): Max retries exceeded with url: /web/query.axd?line=1&type=whiteboard&NegativeScrap=1&units=3&split=job&machines=E0CFAE3D-74EF-0579-8C90-E3D00F56AC70&format=json&start=20180528T060000.000 (Caused by : [Errno 11004] getaddrinfo failed)

I tried using cURL on the same URL with additional arguments and basic auth over HTTPS, and its giving the proper output.

Now I am doubting that this URL itself might not be a API URL that the REST Input App needs..

What are your views?

Regards,
Anirban.

0 Karma

Damien_Dallimor
Ultra Champion

If you google "errno 11004 getaddrinfo failed" you will see that you have hostname resolution errors.

0 Karma

anirbandasdeb
Path Finder

@Damien Dallimore yes, I did that and double checked the settings in the app, with the same error every time.

I am also using this exact same URL+arguments with cURL [basic auth over https] and that is responding fine.

Any methods on confirming if a given URL is actually an API which is REST compliant?
Also, how does the REST API Modular Input App behave if an URL is not an API URL?

The tool itself has very sketchy documentation and the company does not say much about its workings. But I will also try to get information about this.

0 Karma

Damien_Dallimor
Ultra Champion

errno 11004 getaddrinfo failed : you have a DNS error.

This is at the operating system level.

The hostname can not be resolved.

perhaps you are misconfiguring your rest stanza.

please post your full rest stanza for the community to assist in troubleshooting.

0 Karma

anirbandasdeb
Path Finder

what exactly is the rest stanza?

0 Karma

Damien_Dallimor
Ultra Champion

when you setup your rest data input , it gets saved to an inputs.conf file in a [rest] stanza. Search for it under SPLUNK_HOME/etc/*

what does it look like ?

if we have some information to look at , we may be able to help you resolve your operating systems dns lookup failures.

such as , perhaps you entered your hostname in your URL incorrectly ?

0 Karma

anirbandasdeb
Path Finder

okay. let me get a hold of that.
I will get back to you on this @Damien Dallimore

0 Karma

Damien_Dallimor
Ultra Champion

Yes , that is exactly what you can use the REST API Modular Input for.

Get Updates on the Splunk Community!

.conf24 | Day 0

Hello Splunk Community! My name is Chris, and I'm based in Canberra, Australia's capital, and I travelled for ...

Enhance Security Visibility with Splunk Enterprise Security 7.1 through Threat ...

 (view in My Videos)Struggling with alert fatigue, lack of context, and prioritization around security ...

Troubleshooting the OpenTelemetry Collector

  In this tech talk, you’ll learn how to troubleshoot the OpenTelemetry collector - from checking the ...