Best Practice for Getting Data from Splunk Instances Into Indexer Cluster

[Fortron]

I have the following setup with Indexer Discovery + Indexer Cluster + Search Head Cluster:

- Deployment Server

- 3 X Indexer + Cluster Manager (Indexer Cluster)

- Search Head Deployer + Search Head (Set-up as part of a SHC for possible future scaling up)

For forwarding logs from Cluster Manager, I referred to: https://docs.splunk.com/Documentation/Splunk/latest/Indexer/Forwardmanagerdata

For forwarding logs from Search Head Cluster nodes, I referred to: https://docs.splunk.com/Documentation/Splunk/latest/DistSearch/Forwardsearchheaddata

I believe forwarding logs from the Deployment Server should be similar to the above.

For indexers belonging to an indexer cluster, I have considered the following:

1. Install UF in each indexer to monitor & forward logs to the indexer cluster (via indexer discovery)

2. Just monitor logs locally and allow each indexer to index its own local logs (without going through the indexer cluster)

3. Configure the indexer to forward the locally monitored logs without indexing, to the indexer cluster. I am not sure if is necessary to ensure that it does not index the same data twice. Unsure on how this would play out.

Option 2 seems to be the easiest to achieve, but ideally I would like all logs to go through the indexer cluster for indexing.

What should be the best practice for forwarding logs from indexers that are part of the indexer cluster?

[Fortron]

I believe it is due to my lack of understanding on how the indexers in an indexer cluster treat locally monitored data versus data forwarded to the indexer cluster. I mistakenly thought that locally monitored logs on each indexer don't get treated the same way as logs that were forwarded to the indexer cluster.

Thank you for pointing out on the infinite loop, I guess this was the issue when I tried to configure the indexer to forward locally monitored data to its own indexer cluster, which made them spew out alot of errors. 

In that case it seems that I should just create an `inputs.conf` on the indexers and monitor whatever I want, as the indexers' logs would get indexed and subsequently replicated, if I'm understanding it correctly. 

Thank you for your help!

[richgalloway · Accepted solution]

While most instance types should forward their logs to the indexers (using outputs.conf), indexers must not do so lest they cause an infinite loop.  By virtue of the fact the indexer is part of the cluster, its logs go through the cluster.

What problem are you trying to solve?