Introspection Replication

[rsec39]

I have a 2 peer cluster and have run into the following with 2nd cluster bundle deployment (version 6.1.2)

guid=3B88C1A4-84E0-4B8D-A6B4-4E6E5A5B71D8. index='_introspection' app='system' is configured for replication but not specified in bundle downloaded from master. Either define this index on master for replication; or specify repFactor=0 on peer to skip replication.;

[cstump_splunk]

One more thing about this splunkd.log error message:

ERROR loader - Failed to download bundle from master, err="App='system' with replicated index='session_end' is neither in the bundle downloaded from master nor managed by local deployment client. Either define this index at the master or specify repFactor=0 on peer to skip replication. 

If you see this message on your search peer and it is causing the indexer to crash, the issue may be with locally defined indexes. If an index is created through the GUI, the index definition is saved in $SPLUNK_HOME/etc/system/local/indexes.conf. Clustering of the Indexers becomes a problem because of these local indexes.

An index that needs to be a clustered index needs to be in the master_apps directory on the Cluster Master. If there are local index definitions on the clustered indexer, remove them and make sure they are defined on the Cluster Master

[mkolkebeck]

By default, Splunk does not enable index replication (except for a handful of indexes, within master-apps/_cluster/default).

Configuring repFactor = auto by default on the cluster-master so that all indexes replicate was causing your same error for me, but not right away. The initial configuration bundle push was successful, but after any new configuration bundle pushes were made, Splunk complained of the same error.

[default]
repFactor = auto

The fix, I've explicitly set _introspection index with repFactor = 0, keeping the default to replicate all other indexes that are created.

[default]
repFactor = auto
[_introspection]
repFactor = 0

[claudio_manig]

Hi - Thanks for the fix, this worked for me even setting it to "auto". But this seems to be a bug, was this reported to splunk support?

[esalesapns2]

I applied "repFactor = 0" for the _introspection index to an instance running SmartStore. If you do that, your Replication and Search factor will never be met, as shown on your Master Node "Indexer Clustering: Status" page.

[mkolkebeck]

I have not tested this - based on others' comments, it's possible the issue is likely not having setting repFactor explicitly defined for the _introspection index. Therefore, explicitly setting this index to repFactor = auto might be the best solution, especially if you [default]:repFactor = auto. And as @claudio.manig suggests, I recommend opening a support case with Splunk on this if you're continuing to have issues.

[rkirkw]

This helped me as well. With 2 indexer clusters, one started acting this way.

Simply added

 [_introspection]
 repFactor = auto

into my $SPLUNK_HOME/etc/master-apps/_cluster/local/indexes.conf and it fixed it.

These splunk bugs are really annoying and make my job twice as hard as it should be.

[aholzer]

I recently upgraded from Splunk 6.0.3 to 6.1.3 and was receiving this error message.

The solution that finally worked for me was going to $SPLUNK_HOME/etc/master-apps/_cluster, making a copy of the default/indexes.conf into local/indexes.conf and adding the below lines at the end:

[_introspection]
repFactor = auto

Remember not to change the indexes.conf in your "default" folder. Always make a copy of any default configuration to local before changing it.

Hope this helps anyone else that encounters this problem.

[nwales]

Did you get this working?

It caused a major outage for me a few months ago, I had to edit the system/default/indexes.conf with repfactor=0 on the indexers.