App/Add-on

[BRFZ]

I have an architecture with a single SH and two indexers. I've installed the Splunk for Microsoft 365 add-on on the search head, so the collected logs are stored in the search head's index, but I want them to be stored on the indexers. 

Here are two other solutions :

- Either I continue with the initial setup and select only one indexer amont the two to be the storage location for both the search head's data and the add-on.
- Or, I set up a new instance for the heavy forwarder on which I install the add-on, and I configure it to forward the indexes to the Indexer.

Which Solution is the best in my case ?
  

[deepakc]

What I would say is we would normally send all the data to both indexers (This is data load balancing , portions of the data get spread across the two indexers and you get better performance that way, this also has nothing to do with data clustering in the real sense - Best preactise. But it’s your choice at the end of the day.

Heavy Forwarders are typically used for Add-ons – full Splunk instance, from there they send data to indexers. Best practice. You should also send the HF internal logs to the indexers – Best Practise.

The SH connects to the indexers and is configured to sends its logs to the indexers, Best Practise.

The below is an outputs.conf example  that sends to both indexers and sends the local Splunk internal logs , you can tune it to just send to one indexer if you want, just remove the second indexer from the group list and uncomment the specific indexer setting.

Add this to your $SPLUNK_HOME/etc/system/local/outputs.conf on the SH - make the changes to reflect your environment names and test. If you’re using a custom separate app for outputs.conf then add to that. Restart the SH Splunk. You can do the same on a HF.

NOTE:(The new internal indexes _ds*, so these need to be created on the indexer's if you are using the latest versions of Splunk) .
NOTE: Ensure firewalls, ports, NTP have been configured and I'm assuming your not using TLS - thats another subject. 

outputs.conf  Example

[indexAndForward]
index = false

[tcpout]
defaultGroup = <my_group_name_indexers>
forwardedindex.filter.disable = true
indexAndForward = false
forwardedindex.2.whitelist = (_audit|_internal|_introspection|_telemetry|_metrics|_metrics_rollup|_configtracker|_dsclient|_dsphonehome|_dsappevent)

[tcpout:my_group_name_indexers]
#Remove the second indexer if you only want to send to one indexer
server = mysplunk_indexer1:9997, mysplunk_indexer2:9997

#This is only for one indexer receiver
#[tcpout-server://mysplunk_indexer1:9997]

[BRFZ]

Thank you for your assistance and your response 😀

[deepakc]

You should not be instaling indexes on the SH - this is just for search purposes - data is stored on on the indexers and your indexes to should be set there .

From your comment "so the collected logs are stored in the search head's index"... this is not the way

The App/Add-on contain knowledge objects which are used for things like dashboards and parsing search time data, the TA only should be installed onto the indexer or Heavy forward if that is where the data is sent to first.

As to your comment "I set up a new instance for the heavy forwarder on which I install the add-on, and I configure it to forward the indexes to the Indexer."  This is the way forward. 

[BRFZ]

Yes, I know that the search head is not for storing indexes and data, but i've seen that there is also a best practice of forwarding indexes of the search to the indexer layer. Since I don't have an indexer cluster, I need to choose only one indexer among the two. That's why I'm looking for the most suitable method between the two methods I've proposed.

[PickleRick]

No. You can define a output group and load-balance your event between both indexers. They don't have to be cluster members.

[BRFZ]

And in the case of installing and configuring a forwarder to send collected data to the indexers, if the indexers are not in a cluster, is it possible to configure it to send data to both indexers simultaneously ? 

[deepakc]

The forwarders (UF) or HF have inbuilt functionality to send the data to both, so as long as you configure in the outputs.conf, the group names of the servers.

See the section "Configure load balancing on a universal forwarder with outputs.conf"
https://docs.splunk.com/Documentation/Forwarder/9.2.1/Forwarder/Configureforwardingwithoutputs.conf

See this document for how autoload balancing works
https://docs.splunk.com/Documentation/SplunkCloud/9.1.2312/Forwarding/Setuploadbalancingd