Deployment Architecture

Why does the collect command does not work for search head/indexer cluster and I received the following error "event for unconfigured/disabled/deleted"?

highsplunker
Contributor

Hi!

  1. i have an indexer on server_A and a search head on server_B
  2. there is an index=test_ind on server_A
  3. i run a search on server_B (search head) to collect some data to test_in to server_B
    but
    1) i get this error:

    Received event for unconfigured/disabled/deleted index=test_ind with source="source::/opt/splunk/var/spool/splunk/3120f8647b3740cb_events.stash_new"
    host="host::server_B"
    sourcetype="sourcetype::stash".
    So far received events from 11 missing index(es).
    2) and no data collected to test_ind

Note: When I run collect command from the same Splunk instance where test_ind is located, everything is fine; the data is collected.

0 Karma
1 Solution

highsplunker
Contributor

Guys, solved.
The problem was that somehow the forwarding from search head (server_B) to indexer (server_B) Was Broken. I'm not sure, but I did something bad to my deployment server (3rd Splunk instance).

So I justed needed to put "https://serer_A:9997" on my search head via web interface (Settings -> Forwarding ... -> new)
That's it.

Note. Don't make my mistake. If you indicate on your search head which server you want as a peer (in Settings -> Distributed Search) that means FOR SEARCH, NOT COLLECT / INDEX your data. For collecting / indexing your data make sure your forwarding configurations are Ok (via web interface as I described, or via outputs.conf file).

View solution in original post

0 Karma

highsplunker
Contributor

Guys, solved.
The problem was that somehow the forwarding from search head (server_B) to indexer (server_B) Was Broken. I'm not sure, but I did something bad to my deployment server (3rd Splunk instance).

So I justed needed to put "https://serer_A:9997" on my search head via web interface (Settings -> Forwarding ... -> new)
That's it.

Note. Don't make my mistake. If you indicate on your search head which server you want as a peer (in Settings -> Distributed Search) that means FOR SEARCH, NOT COLLECT / INDEX your data. For collecting / indexing your data make sure your forwarding configurations are Ok (via web interface as I described, or via outputs.conf file).

0 Karma

Rob2520
Communicator

Collect command is mainly used to copy data from one index to other. Assuming two indexes are configured properly on indexer(s), and search peer(s) is set on your search head, you can use the following syntax:

index=foo | ... | collect index=bar

Usually the below errors occur when index is not created on indexer(s):

"Received event for unconfigured/disabled/deleted index=test_ind with source="source::/opt/splunk/var/spool/splunk/3120f8647b3740cb_events.stash_new"

If you see the same error again, make sure the index you want to copy to is created properly and do a rolling restart of your clusters peers(indexers).

Hope this helps.

highsplunker
Contributor

Guys, one thing I forgot to add: it worked perfectly, but broke suddenly yesterday.
Of course, we restarted both servers. Not helped.
Any other idea?

0 Karma

highsplunker
Contributor

My guess - it looks like some directory is overfull with files, but I cannot figure out which one...

0 Karma
Get Updates on the Splunk Community!

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...