a few tips on calling REST endpoints

[V_at_Splunk]

Not technically a question, but pretty sure will be helpful to many. If not helpful to you, please don't upvote.

---

What Is REST?

REST is basically:

• a scheme to name and organize endpoints, where an endpoint is a server URI;
• a convention for client to pass GET/POST/DELETE HTTP requests to an endpoint;
• a way to bind server handlers to endpoints, to respond to client requests with a bunch of XML

In other words, YA technology roughly along the lines of CORBA, XML-RPC, SOAP, etc.

Basic Usage

• to create object foo of type widget: services/.../widgets -d name=foo
• to list object foo of type widget: services/.../widgets/foo
• to edit object foo of type widget: services/.../widgets/foo -d someAttr=someVal -X POST
• to delete object foo of type widget: services/.../widgets/foo -X DELETE

-X POST and -X DELETE are options to curl(1); respective alternate forms are --post and --delete.

How to Force App Context

Prepend path with /servicesNS/nobody/your-appname/ instead of just /services.

(Useful if you want the REST call to "come from" a particular app.)

Logging

As of 6.0, tracing to splunkd.log may be requested by setting the special logger REST_Calls to DEBUG.

All attempted calls will be traced, regardless of their eventual success.

For each call, we'll trace:

• app name (search, system, unix, ...)
• HTTP method (GET, POST, ...)
• REST path (data/indexes, deployment/server/clients, ...)
• custom action (countRecentDownloads, ...): if applicable
• the special id caller argument (for data/indexes/eendex, that'd be eendex😞 if any
• remaining caller arguments: if any

Example (see below for explanation of kerl😞

$ kerl data/indexes/foo -d maxMemMB=6

gives

10-03-2013 15:12:31.585 -0700 DEBUG REST_Calls - app=search POST data/indexes/foo id=foo: maxMemMB -> [6]

For result code (200, 409, ...) and response elapsed time, you'll need to look at splunkd_access.log

Calling via curl(1), then Sifting through Results to "Separate Wheat from Chaff", Is a Hassle. Help?

The following bash script, known to work also under Windows via MSYS or Cygwin, may be just the thing. To get usage anytime, just invoke without any arguments, like so (say we named the script kerl😞

$ kerl

USAGE: /home/v/kerl [-v] [--bare] [--app <appName>] [-u <username>] [-p <password>]

                    [<IP|hostname> <port>] <REST path> [REST_arg_0 ... REST_arg_N]

                    [curl options like --get, -i, ...]

Some sample usages:

• On same host as the Splunk server, default username + password, just get index names. (Note that the script is smart enough to pick up your web.conf/[settings]/mgmtPort setting.)

  $ kerl --bare data/indexes
  <title>_audit</title>
  <title>_blocksignature</title>
  <title>_internal</title>
  <title>_thefishbucket</title>
  <title>history</title>
  <title>main</title>
  <title>os</title>
  <title>sample</title>
  <title>splunklogger</title>
  <title>summary</title>
  

• Same, but also show the actual curl(1) command being generated & invoked
  

  
  
  $ kerl -v --bare data/indexes

  Running: curl -s -S -k -u admin:changeme https://127.0.0.1:13542/services/data/indexes

  <title>_audit</title>
  <title>_blocksignature</title>
  <title>_internal</title>
  <title>_thefishbucket</title>
  <title>history</title>
  <title>main</title>
  <title>os</title>
  <title>sample</title>
  <title>splunklogger</title>
  <title>summary</title>
  

• As app search, add a serverclass. (Note how all the boilerplate about ACLs, etc etc, is filtered out for you.)
  

  
  
  $ kerl --app search deployment/server/serverclasses -d name=foo
  
  <feed ...>
  
  <title>serverclasses</title>
  
  <entry>
  
  <title>foo</title>
  
      <blacklist-size>0
  
      <currentDownloads>0
  
      <machineTypesFilter>
  
      <repositoryList>
  
          - app_five
  
          - app_four
  
          - app_three
  
      <repositoryLocation>/home/v/inst/bubbles/etc/deployment-apps
  
      <restartSplunkWeb>0
  
      <restartSplunkd>0
  
      <stateOnClient>enabled
  
      <whitelist-size>0
  
  </entry>
  
  </feed>
  
  

• Query remote DC (deployment client), non-default username + password, ask for HTTP header
  

  
  
  $ kerl -u admax -p changethee 10.1.42.3 13542 deployment/client --get -i
  
  HTTP/1.1 200 OK
  
  Connection: Keep-Alive
  
  Date: Thu, 03 Oct 2013 22:03:02 GMT
  
  Expires: Thu, 26 Oct 1978 00:00:00 GMT
  
  Cache-Control: no-store, no-cache, must-revalidate, max-age=0
  
  Content-Type: text/xml; charset=UTF-8
  
  X-Content-Type-Options: nosniff
  
  Content-Length: 3650
  
  Vary: Authorization
  
  X-Frame-Options: SAMEORIGIN
  
  Server: Splunkd
  
  
  
  <feed ...>
  
  <title>deploymentclient</title>
  
  <entry>
  
  <title>config</title>
  
      <disabled>0
  
      <serverClasses>
  
        <s:list>
  
          <s:item>sc_a:app_one</s:item>
  
          <s:item>sc_a:app_three</s:item>
  
          <s:item>sc_c:app_five</s:item>
  
          <s:item>sc_c:app_one</s:item>
  
        </s:list>
  
      <targetUri>wimpy.splunk.com:13542
  
  </entry>
  
  </feed>
  
  

And here's the script code:

#!/bin/bash

# V 30mar2013

# (c) 2013 by Splunk, Inc.

[ $# -lt 1 ] && echo 'USAGE: [-v] [--bare] [-u <username>] [-p <password>] [<IP|hostname> <port>] <REST path> [REST_arg_0 ... REST_arg_N] [curl options like --get, -i, ...]' >&2 && exit 1
[ -z "$SPLUNK_HOME" ] && echo 'SPLUNK_HOME must be set' >&2 && exit 2

# # # Examples of acceptable REST Paths + Args
# services
# data/indexes/*/roll-hot-buckets
# data/indexes/ -d name=vooble
# data/indexes/vooble
# deployment/server/applications?search=NOT%20serverclasses

# # # Does user want to know exact curl(1) command string?
verbose=false
if [ $1 == '-v' ]; then
verbose=true
shift
fi

# # # Does user want only bare-bones summary of results?
summarizeSedCommand='p'
if [ $1 == '--bare' ]; then
# what this sed script does: only print a line, if the immediately preceding line contained the 'entry' begin-tag.
summarizeSedCommand='x; /<entry>/ {x;p;x}'
shift
fi

# # # Decide auth.
username='admin'
if [ $1 == '-u' ]; then
username="$2"
shift 2
fi

password='changeme'
if [ $1 == '-p' ]; then
password="$2"
shift 2
fi

auth="$username:$password"

# # # Decide host + port.
if [[ $# -ge 3 && $1 =~ ^[[:alnum:]-]?[[:alnum:]\.-]*[[:alnum:]-]?$ && $2 =~ ^[1-9][0-9]*$ && $2 -ge 1 && $2 -le 65535 ]]; then
# Query remote splunkd?
mgmtHostPort="$1:$2"
shift 2
else
# Query local splunkd?
mgmtHostPort=`awk '/^mgmtHostPort/ {print $3}' $SPLUNK_HOME/etc/system/local/web.conf 2>/dev/null`
# server.conf corrupt?
[ -z $mgmtHostPort ] && mgmtHostPort='127.0.0.1:8089'
fi

# # # Decide REST path.
path=${1#/} # delete leading '/' if found
shift # $@ will now contain just REST args
[ ${path:0:8} != 'services' -a ${path:0:10} != 'servicesNS' ] && path="services/$path"

# # # Combine host+port and REST path, to get URL.
url="https://$mgmtHostPort/$path"

# # # Compose output-editing script.
removeSingleLineA='/This is to override browser formatting|<link href|<updated>|"loadTime"|<opensearch:|<generator |xml-stylesheet |xml version=/ d'
removeSingleLineB='/<s:messages\/>|<content |<\/content>|<s:dict>|<\/s:dict>|<id>http/ d'
removeRange='/<author>/,/<\/author>/ d;/<s:key name="eai:acl">/,/^ <\/s:key>/ d;/<s:key name="eai:attributes">/,/^ <\/s:key>/ d'
editWithinLine='/<feed / s/ xmlns=.*$/ ...>/;s/s:key name=//;s/<\/s:key>//;s/"//g;/\/>$/ {s/</- /;s/\/>//}'
removeBlankLine='/^[ ]+$/ d'

# # # Run.
$verbose && echo -e "\n\tRunning:\tcurl $@ -s -S -k -u $auth ${url//%20/ }\n" >&2
curl $@ -s -S -k -u $auth $url | sed -r "$removeSingleLineA;$removeSingleLineB;$removeRange;$editWithinLine;$removeBlankLine" | sed -n "$summarizeSedCommand"

[piebob · Accepted solution]

'answering' this so it stops showing up in the 'double points' list.