After building a deployment and a heavy forwarder on one server we seem to be having issues when we point the universal forwarders to the heavy forwarder. We are new to Splunk 6.3.1 and are not sure if there have been any changes in how to do this?
Overview:
We spun up two new Splunk heavy forwarders for a new company and need both of them to forward logs to our indexers. One of the new heavy forwarders will also act as a deployment server. When setting them up, my stanza are as follows;
Deployment app in Splunk\etc\deployment-apps\App1\default
(outputs.conf)-
[tcpout]
defaultGroup = lb_group
disabled = false
heartbeatFrequency = 300
[tcpout:lb_group]
server = HF1.com:9997, HF2.com:9997
autoLB = true
disabled = false
HF config: Splunk\etc\system\local
(outputs.conf)
[syslog:my_syslog_group]
#FWD logs to an IDS
disabled = false
server = 10.10.10.10:514
type = udp
sendCookedData = false
[tcpout]
defaultGroup = lb_group
disabled = false
[tcpout:lb_group]
server = idx01.com:9997, idx02.com:9997, idx03.com:9997
autoLB = true
disabled = false
Any input will help,
thank you in advance!
This should work. Just note that the HF that runs as the DS cannot be a member of itself. So you will only be able to deploy the config to one of the HFs, not both.
What problem do you see when your UF are pointing to HF?
I do not see any data coming into our indexers when this is set however, i do see the UF's getting the deployment app that points them to the HF's.
When i change the app to send to the indexers to bypass the HF's i see all the data. I don't understand if the HF needs any addition stanzas set to listen for these connections?
Check if this is been set on HF
Set up receiving with the configuration file
You can enable receiving on your Splunk Enterprise instance by configuring inputs.conf in $SPLUNK_HOME/etc/system/local. To configure a universal forwarder as an intermediate forwarder (a forwarder that functions also as a receiver), use this method.
To enable receiving, add a [splunktcp] stanza that specifies the receiving port. In this example, the receiving port is 9997:
[splunktcp://9997]
disabled = 0
This should work. Just note that the HF that runs as the DS cannot be a member of itself. So you will only be able to deploy the config to one of the HFs, not both.