What is an example of what the outputs.conf file would look like on a universal forwarder in an index clustered environment?

[Jarohnimo]

Can someone please provide an example of what the outputs.conf file would look like on a universal forwarder in an index clustered environment?

For example: 1 sh, 2 indexers, 1 clustering Master, 4 nodes with universal forward ready to send data once the setup is complete.

Rep factor 2, search factor 2

1) idx1:9997
2) idx2:9997
3) clustermaster:8089

I've been searching Splunk documentation, but it only provides examples for load balancing forwarders.

Can someone please provide an example of what the outputs.conf file should look like?

[nareshinsvu · Accepted solution]

outputs.conf - if you want to redirect to only specific indexer

[tcpout]
defaultGroup = default-autolb-group
[tcpout:default-autolb-group]
server = idx1:9997
[tcpout-server://idx1:9997]

Else, if you want to discover your indexers through Cluster Master, use below settings - This will help forwarders to route to second indexer if one goes down:

On your Cluster Master's server.conf:

[indexer_discovery]
pass4SymmKey = "create new key and use the same in forwarders outputs.conf"
[clustering]
forwarder_site_failover = site1:site2, site2:site1

On your Forwarder's output.conf:

[indexer_discovery:clustermaster]
pass4SymmKey = "use same key mentioned in your master"
master_uri = https://clustermaster:8089
[tcpout:clustermastergroup]
indexerDiscovery = clustermaster
useACK = true
[tcpout]
defaultGroup = clustermastergroup

On your Forwarder's server.conf:
[general]
site = site1

[Jarohnimo]

Hi Naresh,

Thank you for your response and your assistance is appreciated. so i don't see that it's sending data to the index cluster i created. Via cluster master i deployed index.conf via master-apps, _cluster, local

It then created indexes in the specified location of the index.conf file (everything looks good so far) however on the cluster master page it doesn't show the newly created index, so i'm thinking that's problem #1 Why isn't it showing the new index that the cluster master just created on the peers?

Moving on..

Index cluster is up and running, healthy and replicating _internal indexes. I've added an outputs.conf to one of my web server's universal forwarders "etcs\system\local\" directory with the information below and then I restarted the forwarder

[indexer_discovery:clustermaster]
pass4SymmKey = mypassword
master_uri = https://clustermaster:8089
[tcpout:clustermastergroup]
indexerDiscovery = clustermaster
useACK = true
[tcpout]
defaultGroup = clustermastergroup

My server.conf file for my index cluster master:

[general]
serverName = clustermaster
pass4SymmKey = $7$VDinTNOJp0GCcK0jj8fYCQoxQW6+p3exc2PtgRIEek5OTErTR9+q5g==
sessionTimeout = 1000d

[sslConfig]
sslPassword = $7$6o4579kYGK8VotDH9I5VFy0ly48OdYWJ3jnmvv8tKTFPIUdUebd38w==

[lmpool:auto_generated_pool_download-trial]
description = auto_generated_pool_download-trial
quota = MAX
slaves = *
stack_id = download-trial

[lmpool:auto_generated_pool_forwarder]
description = auto_generated_pool_forwarder
quota = MAX
slaves = *
stack_id = forwarder

[lmpool:auto_generated_pool_free]
description = auto_generated_pool_free
quota = MAX
slaves = *
stack_id = free

[license]
master_uri = https://LicenseMaster:8089

[clustering]
cluster_label = my_cluster
mode = master
pass4SymmKey = $7$497Zb7a04lOvgYxtdzmIiTdcmHomDYYA7TRypAx+LcFwcUXOKz+ovFMHmeA=
replication_factor = 2
search_factor = 1

[indexer_discovery]
pass4SymmKey = $7$5o6HjfUbtuiigSL4yEcVGs6CT8zSCtin+4l+NyTCkWTKF2hLCV7WfZMEVKg=
indexerWeightByDiskCapacity = true

A few things to note. I have a standalone test environment and only used one index called "myindex" Using the deployment server I distributed my apps and all forwarders would send it's data to that stand-alone server to myindex. On the index cluster nodes, i deployed an index.conf file that created the raw index db's for myindex.

When I deleted the original outputs.conf file from the webserver and replaced it with the new one specified above i noticed the forwarders stop sending data to the stand-alone server (GOOD that part is what i wanted) however no data has been sent to the index cluster

I restarted the universal forwarder and I expected to see it forward data to my index cluster but no data has returned. Can you tell me what I've done wrong? did I miss a step somewhere?
.

[nareshinsvu]

Did you use the same passkey for the discovery?

pass4SymmKey = mypassword

Did you define the site ids in your indexer and forwarders? and below setting in CM's server.conf?

[clustering]
forwarder_site_failover = site1:site2, site2:site1

[Jarohnimo]

Ok, I just added the site ID's as you specified still no luck. I still don't see the newly created index showing up on the master nodes clustering screen. It only shows _audit and _internal.

splunk|splunk is owner of splunk directory files, I believe root is the owner of the /opt/indexes/myindex could that be the problem? I'm new to linux and clustering so any help is appreciated

So my problems are

1) though the cluster master was able to create the new index with index.conf and distribute the index , the index still doesn't show up under list of index via the index clustering link on the cluster master

2) though the server is set up as you described above something doesn't seem to be configured correctly. what are some good troubleshooting steps to try and pinpoint the exact issue?

[nareshinsvu]

1) Yes, indexes will show up on the indexer cluster once data goes into it. Which is obviously not working for you at the moment.

2) Can you try reverting all your changes (I mean don't use indexer discovery). You can try basic forwarding as a first step. Can you tell your forwarder to forward data only to one specific index? here there is no need to use any pass key or site ids. if this fails, then there should be some firewall issue with the ports.

outputs.conf - if you want to redirect to only specific indexer

[tcpout]
defaultGroup = default-autolb-group
[tcpout:default-autolb-group]
server = idx1:9997
[tcpout-server://idx1:9997]

[Jarohnimo]

I got it working,

My problem wasn't ports or the security key. It was the fact the my server.conf cluster master label was set to: my_cluster

and the outputs.conf referenced the example set under the indexer discover article here:
https://docs.splunk.com/Documentation/Splunk/7.3.0/Indexer/indexerdiscovery

[indexer_discovery:master1]
pass4SymmKey = my_secret
master_uri = https://10.152.31.202:8089

[tcpout:group1]
autoLBFrequency = 30
forceTimebasedAutoLB = true
indexerDiscovery = master1
useACK=true

[tcpout]
defaultGroup = group1

Once i made both outputs.conf and server.conf files match the same information for cluster label everything worked as expected "ingesting, replication".

If you can edit your original post to include this information i'll accept it as the answer. Thank you!

[nareshinsvu]

I didn't get you. outputs.conf of forwarders doesn't have an entry for cluster master label. Only cluster master URI is mentioned.

Based on the URI, indexer discovery happens on the master and of-course your cluster lable is defined between the indexers and the cluster master irrespective of your forwarders configuration/dependency

[Jarohnimo]

So in my outputs.conf for my UFs

[Indexer_discovery:master1]

[tcpoutput:group1]
IndexerDiscovery=master1

Don't know if this works in combination with cluster label in clusters server.conf
But I made the label master1 (to match) and everything just worked afterward. i also commented out multisite references and rekeyed pass4 on cM and indexers.

[nareshinsvu]

@Jarohnimo

Great that your issue is resolved. Can you accept my response as an answer if it helped?

To my knowledge, there is no direct reference of cluster label usage in the indexer discovery. cluster label is only used between the master and indexers. But no where in the forwarders.

Infact my initial response with all the configurations is from my working environment.