Hi,
Is it possible to query external systems (non-Splunk) with the curl command provided in the JKats toolkit? What is the "data" option for" Posting?
Hi a212830,
The command IS intended to allow you to query external systems.
Sorry for the limited documentation. I'm working on documentation as I add commands.
... | curl post false domain.com/endPoint
... | curl get false domain.com/endPoint
... | curl post true domain.com/someSSLEndpoint
... | curl get true domain.com/someSSLendPoint
The true or false adds https:// or http://, respectively. Here's syntax for actually sending data through to a nonssl endpoint:
... | curl post false domain.com/endPoint '{"key":"value"}' user password
Or if there's no user or password required
... | curl post false domain.com/endPoint '{"key":"value"}'
Hi a212830,
The command IS intended to allow you to query external systems.
Sorry for the limited documentation. I'm working on documentation as I add commands.
... | curl post false domain.com/endPoint
... | curl get false domain.com/endPoint
... | curl post true domain.com/someSSLEndpoint
... | curl get true domain.com/someSSLendPoint
The true or false adds https:// or http://, respectively. Here's syntax for actually sending data through to a nonssl endpoint:
... | curl post false domain.com/endPoint '{"key":"value"}' user password
Or if there's no user or password required
... | curl post false domain.com/endPoint '{"key":"value"}'
Since API's can be a bit wild in their behavior, I dont want to get into the process of transforming the data. There's too many variables...
Instead I'd rather leave it up to you to use | rex field=curl_output "", spath, xpath, extract,, or xmlkv, etc. after the curl command. If you can imagine... your API may return JSON, another might return XML, another plain text, another binary, another broken JSON, another bad xml, etc etc etc. It's too many scenarios for me to anticipate in code, and it's easier for me to put the onus on you to extract what you need from your api.
All that being said... i found a couple more bugs and squashed them in release 0.05.
Understood. Thanks! Great utility!
Ok so this wasnt a true "generating" command and required events prior to it.
I've fixed that in release 0.04 of the toolkit.
Now you can use it with or without prior search results in the pipeline.
| curl post false domain.com/endPoint Will work now
However I believe there will also be a need to use data from the search pipeline. Lets say you have an event that generates JSON which you want to then post to an api. Something like JSONfield='{"key":"value"}'. You can now specify the option 'streaming=true' and then the placeholder for data will look for the field of that name. Example:
|makeresults count=1 | eval jsonData="{'name':'tester','value':'testing'}" | curl post true localhost:80/endPoint jsonData streaming=true
Furthermore, this allows you to make a post/get per event in the pipeline. This search would make 10 posts because makeresults will produce 10 events with the same jsonData field:
|makeresults count=10 | eval jsonData="{'name':'tester','value':'testing'}" | curl post true localhost:80/endPoint jsonData streaming=true
Please do enjoy!!!
Also please note that your username / password will be stored in the splunk internal indexes because splunk tracks the searches you run and you're putting a user/pass in the search. Finally, the streaming option only enables streaming the data field through. It will not for example, allow you to stream in the user/pass from fields. Although it wouldnt be too difficult to achieve this if it is desired.
Great! Any chance it can format it and respect cr/nl? It's just one big stream, which is ugly. Be nice if it could be separate events.
When it returns the data from the endpoint it will be in a field called "curl_output". That field will contain exactly what the endpoint returned in the request body.
Thanks. This is interesting - always thought that it was something missing from Splunk. I have lots of people that want to query external sites via REST, and either create lookups or use as part of their search.
Anyway, I tried this, and got zero events, but think that it should work:
| curl get false "vlcov57:8581/odata/api/devices?" admin admin
Any suggestions?
Do you have anything before the curl command?
Maybe it works if you add events in the pipe first:
| makeresults count=1 | curl ...
Been a moment since I wrote it
Let me test and get back to you tomorrow. I'm pretty sure I made the logic understand your intention with or without the data payload. Maybe I only debugged with post... Happy to fix and take any enhancement requests as well.
Awesome. Thanks. My queries could get quite complicated - the system that I'm interested creates some long url's.
You'll find my email in the app.conf or in any of the Python files. Send me your requirements and I'll do what I can.