a few tips on calling REST endpoints

V_at_Splunk — Mon, 28 Sep 2020 14:53:17 GMT

Not technically a question, but pretty sure will be helpful to many. If not helpful to you, please don't upvote.

What Is REST?

REST is basically:

a scheme to name and organize endpoints, where an endpoint is a server URI;
a convention for client to pass GET/POST/DELETE HTTP requests to an endpoint;
a way to bind server handlers to endpoints, to respond to client requests with a bunch of XML

In other words, YA technology roughly along the lines of CORBA, XML-RPC, SOAP, etc.

Basic Usage

to create object foo of type widget: services/.../widgets -d name=foo
to list object foo of type widget: services/.../widgets/foo
to edit object foo of type widget: services/.../widgets/foo -d someAttr=someVal -X POST
to delete object foo of type widget: services/.../widgets/foo -X DELETE

-X POST and -X DELETE are options to curl(1); respective alternate forms are --post and --delete.

How to Force App Context

Prepend path with /servicesNS/nobody/your-appname/ instead of just /services.

(Useful if you want the REST call to "come from" a particular app.)

Logging

As of 6.0, tracing to splunkd.log may be requested by setting the special logger REST_Calls to DEBUG.

All attempted calls will be traced, regardless of their eventual success.

For each call, we'll trace:

app name (search, system, unix, ...)
HTTP method (GET, POST, ...)
REST path (data/indexes, deployment/server/clients, ...)
custom action (countRecentDownloads, ...): if applicable
the special id caller argument (for data/indexes/eendex, that'd be eendex😞 if any
remaining caller arguments: if any

Example (see below for explanation of kerl😞

$ kerl data/indexes/foo -d maxMemMB=6

gives

10-03-2013 15:12:31.585 -0700 DEBUG REST_Calls - app=search POST data/indexes/foo id=foo: maxMemMB -> [6]

For result code (200, 409, ...) and response elapsed time, you'll need to look at splunkd_access.log

Calling via `curl(1)`, then Sifting through Results to "Separate Wheat from Chaff", Is a Hassle. Help?

The following bash script, known to work also under Windows via MSYS or Cygwin, may be just the thing. To get usage anytime, just invoke without any arguments, like so (say we named the script kerl😞



$ kerl

USAGE: /home/v/kerl [-v] [--bare] [--app <appName>] [-u <username>] [-p <password>]

                    [<IP|hostname> <port>] <REST path> [REST_arg_0 ... REST_arg_N]

                    [curl options like --get, -i, ...]

Some sample usages:

On same host as the Splunk server, default username + password, just get index names. (Note that the script is smart enough to pick up your web.conf/[settings]/mgmtPort setting.)

$ kerl --bare data/indexes
<title>_audit</title>
<title>_blocksignature</title>
<title>_internal</title>
<title>_thefishbucket</title>
<title>history</title>
<title>main</title>
<title>os</title>
<title>sample</title>
<title>splunklogger</title>
<title>summary</title>

Same, but also show the actual curl(1) command being generated & invoked
```
$ kerl -v --bare data/indexes
```
Running: curl -s -S -k -u admin:changeme https://127.0.0.1:13542/services/data/indexes

<title>_audit</title>
<title>_blocksignature</title>
<title>_internal</title>
<title>_thefishbucket</title>
<title>history</title>
<title>main</title>
<title>os</title>
<title>sample</title>
<title>splunklogger</title>
<title>summary</title>

As app search, add a serverclass. (Note how all the boilerplate about ACLs, etc etc, is filtered out for you.)



$ kerl --app search deployment/server/serverclasses -d name=foo

<feed ...>

<title>serverclasses</title>

<entry>

<title>foo</title>

    <blacklist-size>0

    <currentDownloads>0

    <machineTypesFilter>

    <repositoryList>

        - app_five

        - app_four

        - app_three

    <repositoryLocation>/home/v/inst/bubbles/etc/deployment-apps

    <restartSplunkWeb>0

    <restartSplunkd>0

    <stateOnClient>enabled

    <whitelist-size>0

</entry>

</feed>

Query remote DC (deployment client), non-default username + password, ask for HTTP header



$ kerl -u admax -p changethee 10.1.42.3 13542 deployment/client --get -i

HTTP/1.1 200 OK

Connection: Keep-Alive

Date: Thu, 03 Oct 2013 22:03:02 GMT

Expires: Thu, 26 Oct 1978 00:00:00 GMT

Cache-Control: no-store, no-cache, must-revalidate, max-age=0

Content-Type: text/xml; charset=UTF-8

X-Content-Type-Options: nosniff

Content-Length: 3650

Vary: Authorization

X-Frame-Options: SAMEORIGIN

Server: Splunkd



<feed ...>

<title>deploymentclient</title>

<entry>

<title>config</title>

    <disabled>0

    <serverClasses>

      <s:list>

        <s:item>sc_a:app_one</s:item>

        <s:item>sc_a:app_three</s:item>

        <s:item>sc_c:app_five</s:item>

        <s:item>sc_c:app_one</s:item>

      </s:list>

    <targetUri>wimpy.splunk.com:13542

</entry>

</feed>

And here's the script code:



#!/bin/bash

# V 30mar2013

# (c) 2013 by Splunk, Inc.

[ $# -lt 1 ] && echo 'USAGE: [-v] [--bare] [-u <username>] [-p <password>] [<IP|hostname> <port>] <REST path> [REST_arg_0 ... REST_arg_N] [curl options like --get, -i, ...]' >&2 && exit 1
[ -z "$SPLUNK_HOME" ] && echo 'SPLUNK_HOME must be set' >&2 && exit 2

# # # Examples of acceptable REST Paths + Args
# services
# data/indexes/*/roll-hot-buckets
# data/indexes/ -d name=vooble
# data/indexes/vooble
# deployment/server/applications?search=NOT%20serverclasses

# # # Does user want to know exact curl(1) command string?
verbose=false
if [ $1 == '-v' ]; then
verbose=true
shift
fi

# # # Does user want only bare-bones summary of results?
summarizeSedCommand='p'
if [ $1 == '--bare' ]; then
# what this sed script does: only print a line, if the immediately preceding line contained the 'entry' begin-tag.
summarizeSedCommand='x; /<entry>/ {x;p;x}'
shift
fi

# # # Decide auth.
username='admin'
if [ $1 == '-u' ]; then
username="$2"
shift 2
fi

password='changeme'
if [ $1 == '-p' ]; then
password="$2"
shift 2
fi

auth="$username:$password"

# # # Decide host + port.
if [[ $# -ge 3 && $1 =~ ^[[:alnum:]-]?[[:alnum:]\.-]*[[:alnum:]-]?$ && $2 =~ ^[1-9][0-9]*$ && $2 -ge 1 && $2 -le 65535 ]]; then
# Query remote splunkd?
mgmtHostPort="$1:$2"
shift 2
else
# Query local splunkd?
mgmtHostPort=`awk '/^mgmtHostPort/ {print $3}' $SPLUNK_HOME/etc/system/local/web.conf 2>/dev/null`
# server.conf corrupt?
[ -z $mgmtHostPort ] && mgmtHostPort='127.0.0.1:8089'
fi

# # # Decide REST path.
path=${1#/} # delete leading '/' if found
shift # $@ will now contain just REST args
[ ${path:0:8} != 'services' -a ${path:0:10} != 'servicesNS' ] && path="services/$path"

# # # Combine host+port and REST path, to get URL.
url="https://$mgmtHostPort/$path"

# # # Compose output-editing script.
removeSingleLineA='/This is to override browser formatting|<link href|<updated>|"loadTime"|<opensearch:|<generator |xml-stylesheet |xml version=/ d'
removeSingleLineB='/<s:messages\/>|<content |<\/content>|<s:dict>|<\/s:dict>|<id>http/ d'
removeRange='/<author>/,/<\/author>/ d;/<s:key name="eai:acl">/,/^ <\/s:key>/ d;/<s:key name="eai:attributes">/,/^ <\/s:key>/ d'
editWithinLine='/<feed / s/ xmlns=.*$/ ...>/;s/s:key name=//;s/<\/s:key>//;s/"//g;/\/>$/ {s/</- /;s/\/>//}'
removeBlankLine='/^[ ]+$/ d'

# # # Run.
$verbose && echo -e "\n\tRunning:\tcurl $@ -s -S -k -u $auth ${url//%20/ }\n" >&2
curl $@ -s -S -k -u $auth $url | sed -r "$removeSingleLineA;$removeSingleLineB;$removeRange;$editWithinLine;$removeBlankLine" | sed -n "$summarizeSedCommand"

Re: a few tips on calling REST endpoints

piebob — Sun, 21 Dec 2014 19:12:12 GMT

'answering' this so it stops showing up in the 'double points' list.

topic Re: a few tips on calling REST endpoints in Getting Data In