Why does the collect command does not work for search head/indexer cluster and I received the following error "event for unconfigured/disabled/deleted"?

[highsplunker]

Hi!

1. i have an indexer on server_A and a search head on server_B
2. there is an index=test_ind on server_A

3. i run a search on server_B (search head) to collect some data to test_in to server_B
   but
   1) i get this error:

   Received event for unconfigured/disabled/deleted index=test_ind with source="source::/opt/splunk/var/spool/splunk/3120f8647b3740cb_events.stash_new"
   host="host::server_B"
   sourcetype="sourcetype::stash".
   So far received events from 11 missing index(es).
   2) and no data collected to test_ind

Note: When I run collect command from the same Splunk instance where test_ind is located, everything is fine; the data is collected.

[highsplunker · Accepted solution]

Guys, solved.
The problem was that somehow the forwarding from search head (server_B) to indexer (server_B) Was Broken. I'm not sure, but I did something bad to my deployment server (3rd Splunk instance).

So I justed needed to put "https://serer_A:9997" on my search head via web interface (Settings -> Forwarding ... -> new)
That's it.

Note. Don't make my mistake. If you indicate on your search head which server you want as a peer (in Settings -> Distributed Search) that means FOR SEARCH, NOT COLLECT / INDEX your data. For collecting / indexing your data make sure your forwarding configurations are Ok (via web interface as I described, or via outputs.conf file).

[Rob2520]

Collect command is mainly used to copy data from one index to other. Assuming two indexes are configured properly on indexer(s), and search peer(s) is set on your search head, you can use the following syntax:

index=foo | ... | collect index=bar

Usually the below errors occur when index is not created on indexer(s):

"Received event for unconfigured/disabled/deleted index=test_ind with source="source::/opt/splunk/var/spool/splunk/3120f8647b3740cb_events.stash_new"

If you see the same error again, make sure the index you want to copy to is created properly and do a rolling restart of your clusters peers(indexers).

Hope this helps.

[highsplunker]

Guys, one thing I forgot to add: it worked perfectly, but broke suddenly yesterday.
Of course, we restarted both servers. Not helped.
Any other idea?

[highsplunker]

My guess - it looks like some directory is overfull with files, but I cannot figure out which one...