When searching via REST API in a distributed search environment, why am I getting error "supplied index 'p_uno' missing"?

[petreb]

Hallo,

I have a setup with 2 indexers and a dedicated search head; the indexes.conf file is defined only on the indexers (they are configured as deployment clients with the search head as the deployment server in order to simplify the administration of the settings).

Searching via REST API always returns error message "supplied index 'p_uno' missing". According to this:
https://answers.splunk.com/answers/334974/rest-api-receiverssimple-supplied-index-missing.html
the solution would be to define the indexes also on the search head, i.e. the indexes.conf from the deployment class directory should be copied in etc/system/local.

The question is, how can I stop the search head from saving locally the indexed data, when the indexes.conf file gives also the physical paths pro index?

Thanks

[jplumsdaine22 · Accepted solution]

Have you set up the indexers as search peers to the search head? It sounds like your search head is only search itself, but it does not have any data.

Have you followed all the steps here: http://docs.splunk.com/Documentation/Splunk/6.3.3/DistSearch/Configuredistributedsearch

[petreb]

 curl -ku user:pass https://localhost:8089/servicesNS/admin/search/search/jobs/export -d search="search 2016/02/26 03:00:01,some_search_string" -d output_mode=csv

[petreb]

exactly, und according to one question I found here (the link is in my OP) the indexes should be defined also on the indexer (which isn't, in my case)

[jplumsdaine22]

What REST endpoint are you accessing? Can you post your full request? The Splunk answer you lined to is talking about a data input, not a search.

Also can you clarify - in your last comment you said that the index p_uno is NOT defined on your indexers. But in your question you said it is ONLY defined on your indexers.

[petreb]

sorry, my fault, I have thought search head and wrote indexers

the indexes are defined ONLY on the indexers and not on the searchhead

curl -k -u user:pass "https://searchhead:8089/services/receivers/simple?source=www&sourcetype=web_event&index=p_uno" -d  "2016/02/26 03:00:01,some_random_string,0,2367,84032"

answer

<msg type="WARN">supplied index 'p_uno' missing</msg>

answer for index main works

<result>
  <field k="_index">
    <value>
      <text>main</text>
    </value>
  </field>
  <field k="bytes">
    <value>
      <text>60</text>
    </value>
  </field>
  <field k="host">
    <value>
      <text>10.134.222.99</text>
    </value>
  </field>
  <field k="source">
    <value>
      <text>www</text>
    </value>
  </field>
  <field k="sourcetype">
    <value>
      <text>web_event</text>
    </value>
  </field>
</result>

basically he doesn't find the p_uno on the searchhead, because the searchhead doesn't have the indexes defined

what happens if I copy the indexes.conf on the searchhead ? will it also save/store data locally ?

thanks for your patience

[jplumsdaine22]

I see - why not use the rest endpoint on the indexer? Then you don't need to create the index on the search head

[petreb]

I suppose you mean

curl -ku user:pass https://localhost:8089/servicesNS/admin/search/search/jobs/export -d search="search 2016/02/26 03:00:01,some_search_string" -d output_mode=csv

this "spreads" the search across the indexers and gives all available results (see below)

many many thanks

"_serial","_time",source,sourcetype,host,index,"splunk_server","_raw"
0,"2016-02-26 03:00:01.000 CET",www,"web_event","10.104.176.7","p_uno","splunk01-indexer","2016/02/26 03:00:01,some_search_string"
0,"2016-02-26 03:00:01.000 CET",www,"web_event","10.104.176.7","p_uno","splunk02-indexer","2016/02/26 03:00:01,some_search_string"
1,"2016-02-26 03:00:01.000 CET",www,"web_event","10.104.176.7","p_uno","splunk02-indexer","2016/02/26 03:00:01,some_search_string"

[petreb]

yes, the indexers are configured as distributed search peers and they are working; the forwarders are configured to send data only to the indexers and the searchhead reads the data (correctly) from both

[jplumsdaine22]

So searching from web UI works but searching from the API fails?