Getting Data In

Best Practice for Getting Data from Splunk Instances Into Indexer Cluster

Fortron
Engager

I have the following setup with Indexer Discovery + Indexer Cluster + Search Head Cluster:

- Deployment Server

- 3 X Indexer + Cluster Manager (Indexer Cluster)

- Search Head Deployer + Search Head (Set-up as part of a SHC for possible future scaling up)

 

For forwarding logs from Cluster Manager, I referred to: https://docs.splunk.com/Documentation/Splunk/latest/Indexer/Forwardmanagerdata

For forwarding logs from Search Head Cluster nodes, I referred to: https://docs.splunk.com/Documentation/Splunk/latest/DistSearch/Forwardsearchheaddata

I believe forwarding logs from the Deployment Server should be similar to the above.

 

For indexers belonging to an indexer cluster, I have considered the following:

1. Install UF in each indexer to monitor & forward logs to the indexer cluster (via indexer discovery)

2. Just monitor logs locally and allow each indexer to index its own local logs (without going through the indexer cluster)

3. Configure the indexer to forward the locally monitored logs without indexing, to the indexer cluster. I am not sure if is necessary to ensure that it does not index the same data twice. Unsure on how this would play out.

Option 2 seems to be the easiest to achieve, but ideally I would like all logs to go through the indexer cluster for indexing.

What should be the best practice for forwarding logs from indexers that are part of the indexer cluster?

 

Labels (4)
0 Karma
1 Solution

richgalloway
SplunkTrust
SplunkTrust

While most instance types should forward their logs to the indexers (using outputs.conf), indexers must not do so lest they cause an infinite loop.  By virtue of the fact the indexer is part of the cluster, its logs go through the cluster.

What problem are you trying to solve?

---
If this reply helps you, Karma would be appreciated.

View solution in original post

Fortron
Engager

I believe it is due to my lack of understanding on how the indexers in an indexer cluster treat locally monitored data versus data forwarded to the indexer cluster. I mistakenly thought that locally monitored logs on each indexer don't get treated the same way as logs that were forwarded to the indexer cluster.

Thank you for pointing out on the infinite loop, I guess this was the issue when I tried to configure the indexer to forward locally monitored data to its own indexer cluster, which made them spew out alot of errors. 

In that case it seems that I should just create an `inputs.conf` on the indexers and monitor whatever I want, as the indexers' logs would get indexed and subsequently replicated, if I'm understanding it correctly. 

Thank you for your help!

0 Karma

richgalloway
SplunkTrust
SplunkTrust

While most instance types should forward their logs to the indexers (using outputs.conf), indexers must not do so lest they cause an infinite loop.  By virtue of the fact the indexer is part of the cluster, its logs go through the cluster.

What problem are you trying to solve?

---
If this reply helps you, Karma would be appreciated.
Got questions? Get answers!

Join the Splunk Community Slack to learn, troubleshoot, and make connections with fellow Splunk practitioners in real time!

Meet up IRL or virtually!

Join Splunk User Groups to connect and learn in-person by region or remotely by topic or industry.

Get Updates on the Splunk Community!

Accelerate Splunk App Development with AI

  Learn how to automate the repetitive parts of your build cycle.   Watch On-Demand   Replay the Accelerate ...

Splunk Agent Observability: Measuring AI Value with Splunk Tokenomics on AWS

The AI landscape is moving faster than ever, and our latest 2026 State of Observability report provides the ...

Enterprise Security Watch Party: What’s New in 8.8 - Recap

Splunk Community HQ hosted a Splunk Enterprise Security Watch Party: What’s New In ES 8.8 last Thursday. I had ...