REST API receivers/simple supplied index missing

[maynardp]

We are injecting events using the receivers/simple REST API and are not able to specify a specific index.

This does not work (index=web_ops)

/services/receivers/simple?source=www&index=web_ops&sourcetype=hockeyapp '<json data>'

and results in this error

<?xml version="1.0" encoding="UTF-8"?>
<response>
  <messages>
    <msg type="WARN">supplied index 'web_ops' missing</msg>
  </messages>
</response>

However, if I change the index to main the URL does work (index=main)

/services/receivers/simple?source=www&index=main&sourcetype=hockeyapp '<json data>'

So I know the URL is properly formatted.
I created a role for this purpose (rest_api) that inherits User and has additional permission of edit_tcp (as required)
This role has access to search the 'web_ops' index and uses the 'web_ops' app as default app.

I also created a User for this purpose as well that is defined to use the defined 'rest_api' role and again has default app of 'web_ops'.

--
I'm guessing the issue is either a permissions issue with the Role and User I setup to allow REST API to post events, or with the REST API being able to find the indexes (this is a distributed environment with search head pools and index clusters using cluster master and deployment server).

Any assistance would be greatly appreciated.

[jkat54 · Accepted solution]

Are you calling the REST API on a search head?

If so, create the index on the search head(s) manually/programatically. By default when you create an index in a distributed environment, it's only created on the indexers/peers.

Also, you need forwarding enabled on the search head(s) to be sure they'd forward the data back to the indexers/peers.

Another option is to call the REST API on the peers/indexers.

Another option is to setup a heavy forwarder specifically for API calls, and then make sure it gets a copy of indexes.conf, as needed.

[japala]

Hi i have the same problem and i tried different things but i am unable to send the data to splunk using the rest api. can you elaborate the steps. thanks

[maynardp]

You would be much better off using the new HTTP Event Collector in the new versions of Splunk (6.3+). We never went beyond our Stage environment with this.

That said - here's the basic steps we used (not enough space for a detailed explanation):

User account that is writing to REST API must have "edit_tcp" permission. We created a new role for this and a new user account specifically for REST API access that uses this role.

NOTE: You should probably also check what other REST end points to which this permission grants access. The account will likely have the ability to change configuration if you do not set more restrictive permissions. We did not bother to test this - so that is just a guess.

HINT: Data must be send as HTTP POST data. if your data has quotes - best to put your data in a file for testing. Make your life much simpler.

Example of sending data using CURL:

curl -D - -k -u USERID --data @POST_DATA_FILE 'https://SERVERNAME_OR_IP:8089/services/receivers/simple?source=SOURCE&index=INDEXNAME&sourcetype=SOURCETYPE'
command breakdown
-D - (dump headers to STDOUT)
-k (insecure - allow connections to SSL sites without certs)
-u USERID
--data
'https:...' (URL to send to your Splunk instance)

https://:/services/receivers/simple (this is the endpoint for injection of events using simple receiver method)

source=SOURCE (give your event a source - I usually use Web or httpRest)

index=INDEX (your user MUST have access to this index or the post will fail and if you don't specify an index, you will inject into the default index.)

sourcetype=SOURCETYPE (some uniq sourcetype that has meaning to you and is not already in use - make searching for new events much simpler)

you will be prompted for the password in this example because I don't put it in the curl command
you should receive a 200 OK message if it was successful
any other response code means you have a problem.
If you see something like the following, it means the index (web_ops in this example output) is not valid or accessible to this user or via this Mgmt UI

<?xml version="1.0" encoding="UTF-8"?>
<response>
  <messages>
    <msg type="WARN">supplied index 'web_ops' missing</msg>
  </messages>
</response>

Once injected - you can view the event using standard splunk search

[japala]

Awesome @maynardp man.. that is a detailed explanation. thank you very much.. i have went through all the steps you mentioned but i missed some of the steps.. i will try to rectify the mistakes i did and i will do it again..

[maynardp]

Much appreciated. I managed to figure out that Search Heads were never intended to be forwarders in a distributed environment and that this was the wrong approach. I was just coming to update my post when I saw your response. Thanks you for the excellent info.

[jkat54]

Many thanks for returning and marking the answer!