Monitor logs on Indexers and forward to another indexer cluster

[mccartneyc]

In my environment, we send everything to our indexer cluster and use data cloning using _TCP_ROUTING on the universal forwarders to send some data to both the indexer cluster and to an intermediate forwarder. The intermediate forwarder sends the cloned data to another company's splunk environment for analysis.

Everything is working as intended, but I can't get the indexers to monitor local /var/log/ logs, index them and also forward them to the intermediate forwarder for forwarding. I looked into the indexing and forwarding option, but that forwards indexed data to the intermediate, when all I want is for the indexer to send it's monitored logs to the cluster and the intermediate.

The reason behind this setup is because of our security requirements with our network and ingesting data from clients.

Is there a way to do the following:
1) UF sends data to both the indexer cluster and the intermediate
2) Monitor /var/log and other log files on the indexers, index locally while also sending those to the intermediate
3) Do the above without forwarding data being received/indexed by the indexers from forwarders to the intermediate

[bandit]

You could install a Universal Forwarder on the same host as the indexer and have it take over the monitoring/forwarding of Splunk indexer logs to all destinations.

I don't have your exact use case, however, I do run Universal Forwarders on my indexers and search heads to monitor things like CPU and OS metrics, without having to worry about restarting the indexers. That way I can restart the universal forwarders on my indexers at any time without impacting the indexer if I want to make changes to what I'm collecting or collection intervals, etc.