topic how does Universal Forwarder work? in Getting Data In

how does Universal Forwarder work?

nanachu — Mon, 16 Dec 2019 06:30:05 GMT

Hi, all

I wonder about Universal Forwarder.

I have to switch master uri of deploymentclient.conf and outputs.conf
because I created new cluster master(new is production environment)

If the switch does not work, I change master uri to original.

Will I lost data during the switchover work?
Or will the UF send the past data when it is change to master node?

If UF sends data to the new cluster master and then turns masteruri to original cluster master,
Does the data that has already been sent can not index in original?

Thank you for helping me.

Re: how does Universal Forwarder work?

richgalloway — Mon, 16 Dec 2019 13:20:06 GMT

Universal Forwarders do not send data to the Master Node. MN is consulted only during Indexer Discovery to get a list of indexers. Data is sent to those indexers.
When you change the MN's URI, the UF will contact the new MN and send data to the indexers it provides.

Re: how does Universal Forwarder work?

gcusello — Mon, 16 Dec 2019 13:27:15 GMT

Hi @nanachu,
at first in deploymentclient.conf there's the address of the Deployment Server that hasn't any relation with the cluster and the Master Node (if you have the Master Node configured also as Deployment Server is an error!).

Now you have an outputs.conf where is addressed the Master Node of an Indexers' Cluster and you want to switch to another Master Node, because you're changing the Indexers' Cluster, is it correct?

If you address the Master Node instead the Seach Peers, means that you're using indexer discovery feature, so you have to follow the configurations described in https://docs.splunk.com/Documentation/Splunk/8.0.1/Indexer/indexerdiscovery :
In the master node's: server.conf:

[indexer_discovery]
pass4SymmKey = my_secret
indexerWeightByDiskCapacity = true

In each forwarder's outputs.conf:

[indexer_discovery:master1]
pass4SymmKey = my_secret
master_uri = https://xxx.xxx.xxx.xxx:8089

[tcpout:group1]
autoLBFrequency = 30
forceTimebasedAutoLB = true
indexerDiscovery = master1
useACK=true

[tcpout]
defaultGroup = group1

Ciao.
Giuseppe

Re: how does Universal Forwarder work?

woodcock — Mon, 16 Dec 2019 15:46:28 GMT

You are mixing terms, settings, and files. The master_uri in deployment_client.conf applies to your Deployment Server, NOT your Cluster Master.

First, let's assume that you really mean Cluster Master. If you are switching ONLY the Cluster Master (AKA Master Node) in server.conf which controls your Indexer bucket replication, then you are NOT at any risk of losing data UNLESS you are using the Indexer Discovery feature in the outputs.conf for your forwarders (most people are not). If you are using this feature AND you mess up AND a forwarder has Splunk rebooted, it will not be able to get a list of Indexers so it will not be able to send data, it will buffer for a while, and then start dropping data. That is a big stack of ifs.

Now, let's assume that you really mean Deployment Server. If you change the Deployment Server in deploymentclient.conf and you mess up, you have created a truly formidable disaster. Once you create a deployment zombie, the only way to get command and control back, is to log back onto that box and manually fix it. Even so, the far more likely problem that you will have is that the initial admin used the set deploy-server CLI command which put the deploymentclient.conf file into $SPLUNK_HOME/etc/system/local/ which means that your change will be ignored by some (maybe all) deployment clients. Again, you must log onto each box and re/move the file from that location before your change will work for that server.