topic Re: Can you query external systems with the curl command in JKats Toolkit? in All Apps and Add-ons

Can you query external systems with the curl command in JKats Toolkit?

a212830 — Mon, 22 Aug 2016 18:44:21 GMT

Hi,

Is it possible to query external systems (non-Splunk) with the curl command provided in the JKats toolkit? What is the "data" option for" Posting?

Re: Can you query external systems with the curl command in JKats Toolkit?

jkat54 — Mon, 22 Aug 2016 22:46:19 GMT

Hi a212830,

The command IS intended to allow you to query external systems.

Sorry for the limited documentation. I'm working on documentation as I add commands.

... | curl post false domain.com/endPoint
... | curl get false domain.com/endPoint
... | curl post true domain.com/someSSLEndpoint
... | curl get true domain.com/someSSLendPoint

The true or false adds https:// or http://, respectively. Here's syntax for actually sending data through to a nonssl endpoint:

... | curl post false domain.com/endPoint '{"key":"value"}' user password

Or if there's no user or password required

... | curl post false domain.com/endPoint '{"key":"value"}'

Re: Can you query external systems with the curl command in JKats Toolkit?

jkat54 — Mon, 22 Aug 2016 22:48:22 GMT

When it returns the data from the endpoint it will be in a field called "curl_output". That field will contain exactly what the endpoint returned in the request body.

Re: Can you query external systems with the curl command in JKats Toolkit?

a212830 — Tue, 23 Aug 2016 00:34:24 GMT

Thanks. This is interesting - always thought that it was something missing from Splunk. I have lots of people that want to query external sites via REST, and either create lookups or use as part of their search.

Anyway, I tried this, and got zero events, but think that it should work:

| curl get false "vlcov57:8581/odata/api/devices?" admin admin

Any suggestions?

Re: Can you query external systems with the curl command in JKats Toolkit?

jkat54 — Tue, 23 Aug 2016 00:45:08 GMT

Let me test and get back to you tomorrow. I'm pretty sure I made the logic understand your intention with or without the data payload. Maybe I only debugged with post... Happy to fix and take any enhancement requests as well.

Re: Can you query external systems with the curl command in JKats Toolkit?

a212830 — Tue, 23 Aug 2016 00:47:42 GMT

Awesome. Thanks. My queries could get quite complicated - the system that I'm interested creates some long url's.

Re: Can you query external systems with the curl command in JKats Toolkit?

jkat54 — Tue, 23 Aug 2016 00:48:45 GMT

Do you have anything before the curl command?

Maybe it works if you add events in the pipe first:

| makeresults count=1 | curl ...

Been a moment since I wrote it

Re: Can you query external systems with the curl command in JKats Toolkit?

jkat54 — Tue, 23 Aug 2016 00:59:16 GMT

You'll find my email in the app.conf or in any of the Python files. Send me your requirements and I'll do what I can.

Re: Can you query external systems with the curl command in JKats Toolkit?

jkat54 — Tue, 23 Aug 2016 02:49:54 GMT

Ok so this wasnt a true "generating" command and required events prior to it.

I've fixed that in release 0.04 of the toolkit.

Now you can use it with or without prior search results in the pipeline.

 | curl post false domain.com/endPoint   Will work now

However I believe there will also be a need to use data from the search pipeline. Lets say you have an event that generates JSON which you want to then post to an api. Something like JSONfield='{"key":"value"}'. You can now specify the option 'streaming=true' and then the placeholder for data will look for the field of that name. Example:

 |makeresults count=1 | eval jsonData="{'name':'tester','value':'testing'}" |  curl post true localhost:80/endPoint jsonData streaming=true

Furthermore, this allows you to make a post/get per event in the pipeline. This search would make 10 posts because makeresults will produce 10 events with the same jsonData field:

 |makeresults count=10 | eval jsonData="{'name':'tester','value':'testing'}" |  curl post true localhost:80/endPoint jsonData streaming=true

Please do enjoy!!!

Re: Can you query external systems with the curl command in JKats Toolkit?

jkat54 — Tue, 23 Aug 2016 02:52:54 GMT

Also please note that your username / password will be stored in the splunk internal indexes because splunk tracks the searches you run and you're putting a user/pass in the search. Finally, the streaming option only enables streaming the data field through. It will not for example, allow you to stream in the user/pass from fields. Although it wouldnt be too difficult to achieve this if it is desired.

Re: Can you query external systems with the curl command in JKats Toolkit?

a212830 — Tue, 23 Aug 2016 11:39:27 GMT

Great! Any chance it can format it and respect cr/nl? It's just one big stream, which is ugly. Be nice if it could be separate events.

Re: Can you query external systems with the curl command in JKats Toolkit?

jkat54 — Tue, 23 Aug 2016 12:45:32 GMT

Since API's can be a bit wild in their behavior, I dont want to get into the process of transforming the data. There's too many variables...

Instead I'd rather leave it up to you to use | rex field=curl_output "", spath, xpath, extract,, or xmlkv, etc. after the curl command. If you can imagine... your API may return JSON, another might return XML, another plain text, another binary, another broken JSON, another bad xml, etc etc etc. It's too many scenarios for me to anticipate in code, and it's easier for me to put the onus on you to extract what you need from your api.

All that being said... i found a couple more bugs and squashed them in release 0.05.

Re: Can you query external systems with the curl command in JKats Toolkit?

a212830 — Tue, 23 Aug 2016 14:32:00 GMT

Understood. Thanks! Great utility!