Why is my Distributed Management Console trying to push a bundle to a newly added search peer which happens to be a standalone indexer?

[lycollicott]

I have a single Distributed Management Console which I have monitoring separated regional indexers like so....

I had everything from Region 1 registered in the DMC first and then I registered the Region 2 standalone indexer and now I see these messages in remote_searches.log on each of my Region 2 clustered indexers.....

INFO StreamedSearch - Streamed search connection terminated: search_id=remote_REGION_1_SEARCHHEAD_123456789, server=REGION_1_SEARCHHEAD, active_searches=1, elapsedTime=0.641, search='litsearch index=_internal "Unable to distribute to peer named REGION_2_INDEXER" | fields keepcolorder=t "*" "_bkt" "_cd" "_si" "host" "index" "linecount" "source" "sourcetype" "splunk_server" | remotetl nb=300 et=1455733920.000000 lt=1455737578.000000 remove=true max_count=1000 max_prefetch=100', savedsearch_name=""

This also occurs in splunkd,log on the DMC.....

WARN DistributedPeerManager - Unable to distribute to peer named REGION_2_INDEXER at uri https://REGION_2_INDEXER :8089 because replication was unsuccessful. replicationStatus Failed failure info: failed_because_BUNDLE_DATA_TRANSMIT_FAILURE

I don't understand why the DMC is trying to push a bundle to the Region 2 indexer.

[ykou_splunk · Accepted solution]

I don't understand why the DMC is trying to push a bundle to the Region 2 indexer.

I think the "bundle push" here refers to the search knowledge objects replication, which is expected, because DMC needs to do ad-hoc search against that indexer to monitor that indexer. Here's the docs talking about what happened: http://docs.splunk.com/Documentation/Splunk/6.3.3/DistSearch/Whatsearchheadssend

Basically, DMC monitors other splunk instances by doing ad-hoc searches (to get historical data from log events and current data from REST endpoints) against the splunk instances being monitored.

In your case, the Region 2 indexer is a distributed search peer of the DMC instance. So, when DMC starts a search, it will send the search knowledge bundles to the Region 2 indexer in order to complete the search.

Please note that the concept of "bundle push" in this context is different from the concept of "app bundle push" or "configuration bundle push". Search knowledge objects bundle push happens when a search head starts a search against it's distributed peers, while "app bundle push" or "configuration bundle push" happens when you want to deploy some apps or configurations (typically from cluster master or deployment server) to some splunk instances.