All Apps and Add-ons

Can you query external systems with the curl command in JKats Toolkit?

a212830
Champion

Hi,

Is it possible to query external systems (non-Splunk) with the curl command provided in the JKats toolkit? What is the "data" option for" Posting?

1 Solution

jkat54
SplunkTrust
SplunkTrust

Hi a212830,

The command IS intended to allow you to query external systems.

Sorry for the limited documentation. I'm working on documentation as I add commands.

... | curl post false domain.com/endPoint
... | curl get false domain.com/endPoint
... | curl post true domain.com/someSSLEndpoint
... | curl get true domain.com/someSSLendPoint

The true or false adds https:// or http://, respectively. Here's syntax for actually sending data through to a nonssl endpoint:

... | curl post false domain.com/endPoint '{"key":"value"}' user password

Or if there's no user or password required

... | curl post false domain.com/endPoint '{"key":"value"}'

View solution in original post

jkat54
SplunkTrust
SplunkTrust

Hi a212830,

The command IS intended to allow you to query external systems.

Sorry for the limited documentation. I'm working on documentation as I add commands.

... | curl post false domain.com/endPoint
... | curl get false domain.com/endPoint
... | curl post true domain.com/someSSLEndpoint
... | curl get true domain.com/someSSLendPoint

The true or false adds https:// or http://, respectively. Here's syntax for actually sending data through to a nonssl endpoint:

... | curl post false domain.com/endPoint '{"key":"value"}' user password

Or if there's no user or password required

... | curl post false domain.com/endPoint '{"key":"value"}'

jkat54
SplunkTrust
SplunkTrust

Since API's can be a bit wild in their behavior, I dont want to get into the process of transforming the data. There's too many variables...

Instead I'd rather leave it up to you to use | rex field=curl_output "", spath, xpath, extract,, or xmlkv, etc. after the curl command. If you can imagine... your API may return JSON, another might return XML, another plain text, another binary, another broken JSON, another bad xml, etc etc etc. It's too many scenarios for me to anticipate in code, and it's easier for me to put the onus on you to extract what you need from your api.

All that being said... i found a couple more bugs and squashed them in release 0.05.

alt text

0 Karma

a212830
Champion

Understood. Thanks! Great utility!

0 Karma

jkat54
SplunkTrust
SplunkTrust

Ok so this wasnt a true "generating" command and required events prior to it.

I've fixed that in release 0.04 of the toolkit.

Now you can use it with or without prior search results in the pipeline.

 | curl post false domain.com/endPoint   Will work now

However I believe there will also be a need to use data from the search pipeline. Lets say you have an event that generates JSON which you want to then post to an api. Something like JSONfield='{"key":"value"}'. You can now specify the option 'streaming=true' and then the placeholder for data will look for the field of that name. Example:

 |makeresults count=1 | eval jsonData="{'name':'tester','value':'testing'}" |  curl post true localhost:80/endPoint jsonData streaming=true

Furthermore, this allows you to make a post/get per event in the pipeline. This search would make 10 posts because makeresults will produce 10 events with the same jsonData field:

 |makeresults count=10 | eval jsonData="{'name':'tester','value':'testing'}" |  curl post true localhost:80/endPoint jsonData streaming=true

Please do enjoy!!!

jkat54
SplunkTrust
SplunkTrust

Also please note that your username / password will be stored in the splunk internal indexes because splunk tracks the searches you run and you're putting a user/pass in the search. Finally, the streaming option only enables streaming the data field through. It will not for example, allow you to stream in the user/pass from fields. Although it wouldnt be too difficult to achieve this if it is desired.

0 Karma

a212830
Champion

Great! Any chance it can format it and respect cr/nl? It's just one big stream, which is ugly. Be nice if it could be separate events.

0 Karma

jkat54
SplunkTrust
SplunkTrust

When it returns the data from the endpoint it will be in a field called "curl_output". That field will contain exactly what the endpoint returned in the request body.

0 Karma

a212830
Champion

Thanks. This is interesting - always thought that it was something missing from Splunk. I have lots of people that want to query external sites via REST, and either create lookups or use as part of their search.

Anyway, I tried this, and got zero events, but think that it should work:

| curl get false "vlcov57:8581/odata/api/devices?" admin admin

Any suggestions?

0 Karma

jkat54
SplunkTrust
SplunkTrust

Do you have anything before the curl command?

Maybe it works if you add events in the pipe first:

| makeresults count=1 | curl ...

Been a moment since I wrote it

0 Karma

jkat54
SplunkTrust
SplunkTrust

Let me test and get back to you tomorrow. I'm pretty sure I made the logic understand your intention with or without the data payload. Maybe I only debugged with post... Happy to fix and take any enhancement requests as well.

0 Karma

a212830
Champion

Awesome. Thanks. My queries could get quite complicated - the system that I'm interested creates some long url's.

0 Karma

jkat54
SplunkTrust
SplunkTrust

You'll find my email in the app.conf or in any of the Python files. Send me your requirements and I'll do what I can.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...