Getting Data In

a few tips on calling REST endpoints

Splunk Employee
Splunk Employee

Not technically a question, but pretty sure will be helpful to many. If not helpful to you, please don't upvote.

What Is REST?

REST is basically:

  • a scheme to name and organize endpoints, where an endpoint is a server URI;
  • a convention for client to pass GET/POST/DELETE HTTP requests to an endpoint;
  • a way to bind server handlers to endpoints, to respond to client requests with a bunch of XML

In other words, YA technology roughly along the lines of CORBA, XML-RPC, SOAP, etc.

Basic Usage

  • to create object foo of type widget: services/.../widgets -d name=foo
  • to list object foo of type widget: services/.../widgets/foo
  • to edit object foo of type widget: services/.../widgets/foo -d someAttr=someVal -X POST
  • to delete object foo of type widget: services/.../widgets/foo -X DELETE

-X POST and -X DELETE are options to curl(1); respective alternate forms are --post and --delete.

How to Force App Context

Prepend path with /servicesNS/nobody/your-appname/ instead of just /services.

(Useful if you want the REST call to "come from" a particular app.)


As of 6.0, tracing to splunkd.log may be requested by setting the special logger REST_Calls to DEBUG.

All attempted calls will be traced, regardless of their eventual success.

For each call, we'll trace:

  • app name (search, system, unix, ...)
  • HTTP method (GET, POST, ...)
  • REST path (data/indexes, deployment/server/clients, ...)
  • custom action (countRecentDownloads, ...): if applicable
  • the special id caller argument (for data/indexes/eendex, that'd be eendex😞 if any
  • remaining caller arguments: if any

Example (see below for explanation of kerl😞

$ kerl data/indexes/foo -d maxMemMB=6


10-03-2013 15:12:31.585 -0700 DEBUG REST_Calls - app=search POST data/indexes/foo id=foo: maxMemMB -> [6]

For result code (200, 409, ...) and response elapsed time, you'll need to look at splunkd_access.log

Calling via curl(1), then Sifting through Results to "Separate Wheat from Chaff", Is a Hassle. Help?

The following bash script, known to work also under Windows via MSYS or Cygwin, may be just the thing. To get usage anytime, just invoke without any arguments, like so (say we named the script kerl😞

$ kerl
USAGE: /home/v/kerl [-v] [--bare] [--app <appName>] [-u <username>] [-p <password>]
[<IP|hostname> <port>] <REST path> [REST_arg_0 ... REST_arg_N]
[curl options like --get, -i, ...]

Some sample usages:

  • On same host as the Splunk server, default username + password, just get index names. (Note that the script is smart enough to pick up your web.conf/[settings]/mgmtPort setting.)
    $ kerl --bare data/indexes
  • Same, but also show the actual curl(1) command being generated & invoked

    $ kerl -v --bare data/indexes

    Running: curl -s -S -k -u admin:changeme


  • As app search, add a serverclass. (Note how all the boilerplate about ACLs, etc etc, is filtered out for you.)

    $ kerl --app search deployment/server/serverclasses -d name=foo
    <feed ...>
    - app_five
    - app_four
    - app_three

  • Query remote DC (deployment client), non-default username + password, ask for HTTP header

    $ kerl -u admax -p changethee 13542 deployment/client --get -i
    HTTP/1.1 200 OK
    Connection: Keep-Alive
    Date: Thu, 03 Oct 2013 22:03:02 GMT
    Expires: Thu, 26 Oct 1978 00:00:00 GMT
    Cache-Control: no-store, no-cache, must-revalidate, max-age=0
    Content-Type: text/xml; charset=UTF-8
    X-Content-Type-Options: nosniff
    Content-Length: 3650
    Vary: Authorization
    X-Frame-Options: SAMEORIGIN
    Server: Splunkd

    <feed ...>

And here's the script code:

# V 30mar2013
# (c) 2013 by Splunk, Inc.

[ $# -lt 1 ] && echo 'USAGE: [-v] [--bare] [-u <username>] [-p <password>] [<IP|hostname> <port>] <REST path> [REST_arg_0 ... REST_arg_N] [curl options like --get, -i, ...]' >&2 && exit 1
[ -z "$SPLUNK_HOME" ] && echo 'SPLUNK_HOME must be set' >&2 && exit 2

# # # Examples of acceptable REST Paths + Args
# services
# data/indexes/*/roll-hot-buckets
# data/indexes/ -d name=vooble
# data/indexes/vooble
# deployment/server/applications?search=NOT%20serverclasses

# # # Does user want to know exact curl(1) command string?
if [ $1 == '-v' ]; then

# # # Does user want only bare-bones summary of results?
if [ $1 == '--bare' ]; then
# what this sed script does: only print a line, if the immediately preceding line contained the 'entry' begin-tag.
summarizeSedCommand='x; /<entry>/ {x;p;x}'

# # # Decide auth.
if [ $1 == '-u' ]; then
shift 2

if [ $1 == '-p' ]; then
shift 2


# # # Decide host + port.
if [[ $# -ge 3 && $1 =~ ^[[:alnum:]-]?[[:alnum:]\.-]*[[:alnum:]-]?$ && $2 =~ ^[1-9][0-9]*$ && $2 -ge 1 && $2 -le 65535 ]]; then
# Query remote splunkd?
shift 2
# Query local splunkd?
mgmtHostPort=`awk '/^mgmtHostPort/ {print $3}' $SPLUNK_HOME/etc/system/local/web.conf 2>/dev/null`
# server.conf corrupt?
[ -z $mgmtHostPort ] && mgmtHostPort=''

# # # Decide REST path.
path=${1#/} # delete leading '/' if found
shift # $@ will now contain just REST args
[ ${path:0:8} != 'services' -a ${path:0:10} != 'servicesNS' ] && path="services/$path"

# # # Combine host+port and REST path, to get URL.

# # # Compose output-editing script.
removeSingleLineA='/This is to override browser formatting|<link href|<updated>|"loadTime"|<opensearch:|<generator |xml-stylesheet |xml version=/ d'
removeSingleLineB='/<s:messages\/>|<content |<\/content>|<s:dict>|<\/s:dict>|<id>http/ d'
removeRange='/<author>/,/<\/author>/ d;/<s:key name="eai:acl">/,/^ <\/s:key>/ d;/<s:key name="eai:attributes">/,/^ <\/s:key>/ d'
editWithinLine='/<feed / s/ xmlns=.*$/ ...>/;s/s:key name=//;s/<\/s:key>//;s/"//g;/\/>$/ {s/</- /;s/\/>//}'
removeBlankLine='/^[ ]+$/ d'

# # # Run.
$verbose && echo -e "\n\tRunning:\tcurl $@ -s -S -k -u $auth ${url//%20/ }\n" >&2
curl $@ -s -S -k -u $auth $url | sed -r "$removeSingleLineA;$removeSingleLineB;$removeRange;$editWithinLine;$removeBlankLine" | sed -n "$summarizeSedCommand"

1 Solution

Splunk Employee
Splunk Employee

'answering' this so it stops showing up in the 'double points' list.

View solution in original post

Splunk Employee
Splunk Employee

'answering' this so it stops showing up in the 'double points' list.

Get Updates on the Splunk Community!

Optimize Cloud Monitoring

  TECH TALKS Optimize Cloud Monitoring Tuesday, August 13, 2024  |  11:00AM–12:00PM PST   Register to ...

What's New in Splunk Cloud Platform 9.2.2403?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.2.2403! Analysts can ...

Stay Connected: Your Guide to July and August Tech Talks, Office Hours, and Webinars!

Dive into our sizzling summer lineup for July and August Community Office Hours and Tech Talks. Scroll down to ...