Heavy Forwarder Load Balancing syslog data to F5 VIPs

[tiaatim]

We are on 7.2.5.1. My outputs is sending incoming Windows logs out to 2 F5 VIPs via a syslog stanza. The data is going out and only ever hits the first vip in the server= line in the stanza

[syslog:test_group]
priority = NO_PRI
server = 10.X.X.1:514,10.X.X.2:514
type = udp

The .1 is receiving all of the data on the F5 and the HF never seems to switch over to the .2 IP.

Any help would be greatly appreciated.

[codebuilder]

Your configuration is not correct for what you're trying to accomplish.

The "server" parameter in the [syslog] stanza takes only a single IP:PORT and is where you define the address to your syslog server (singular).
This parameter is not for load balancing, and is the reason only one of your two defined IP's is receiving data.

https://docs.splunk.com/Documentation/Splunk/8.0.1/Admin/Outputsconf#Syslog_output----

[tiaatim]

Thanks, so do something like this?

[syslog]
defaultGroup = syslog:test_group1, syslog:test_group2
type = udp
priority = NO_PRI

[syslog:test_group1]
server = 10.X.X.1:514

[syslog:test_group2]
server = 10.X.X.2:514

[codebuilder]

It's a bit of a one-off configuration, and I'm not sure your results will be consistent, but this config combo should work. Though I can't be certain you won't get duplicate data.

[syslog]
defaultGroup = syslog:test_group1, syslog:test_group2
type = udp
priority = NO_PRI

[syslog:test_group1]
server = 10.X.X.1:514

[syslog:test_group2]
server = 10.X.X.2:514

[tiaatim]

I tried the above config and unfortunately it didn't work. It still only sent to one IP address. Here is the outputs.conf that I tried:

[syslog]
defaultGroup = syslog:test_group1,syslog:test_group2
type = udp
priority = NO_PRI

[syslog:test_group1]
server = 10.x.x.1:514

[syslog:test_group2]
server = 10.x.x.2:514

[codebuilder]

A number of things could cause that. Did you cycle Splunk after making the config change? Is the F5 using a wildcard on the IP? Have you tried commenting out test_group1 to verify data reaches test_group2?

Again, what you're trying to do is a one-off, and isn't really meant to be configured this way. It may honestly be time to re-think your approach on this.

[tiaatim]

Yes, I did restart Splunk. I didn't comment out test_group1 this time but did before I made this specific change and saw data going to group2. Thanks for your help.

[codebuilder]

You could try the example in the documentation, which will work, and also ensure you don't get duplicate data.

#
# The following example shows how to route events to syslog server
# This is similar to tcpout routing, but DEST_KEY is set to _SYSLOG_ROUTING
#

# 1. Edit $SPLUNK_HOME/etc/system/local/props.conf and set a TRANSFORMS-routing
#    attribute:
[default]
TRANSFORMS-routing=errorRouting

[syslog]
TRANSFORMS-routing=syslogRouting

# 2. Edit $SPLUNK_HOME/etc/system/local/transforms.conf and set errorRouting
#    and syslogRouting rules:
[errorRouting]
REGEX=error
DEST_KEY=_SYSLOG_ROUTING
FORMAT=errorGroup

[syslogRouting]
REGEX=.
DEST_KEY=_SYSLOG_ROUTING
FORMAT=syslogGroup

# 3. Edit $SPLUNK_HOME/etc/system/local/outputs.conf and set which syslog
#    outputs go to with servers or groups:
[syslog]
defaultGroup=everythingElseGroup

[syslog:syslogGroup]
server = 10.1.1.197:9997

[syslog:errorGroup]
server=10.1.1.200:9999

[syslog:everythingElseGroup]
server=10.1.1.250:6666

https://docs.splunk.com/Documentation/Splunk/latest/Admin/Outputsconf?utm_source=answers&utm_medium=...

[codebuilder]

I'm assuming that you are using two F5's for redundancy, but you can configure a single VIP across two clustered F5's (much easier in haproxy), and that would solve your issue.

[tiaatim]

Currently the 2 VIPs are on one F5. We had one VIP going to a 2 IP pool but saw that traffic was only going to a single IP in the pool. So we are now trying 2 VIPs with the 2 IPs in the pool's flipped so the first VIP goes to .3 and .4 in the pool and the second VIP goes to .4 and .3 in the pool. Make sense?

[13tsavage]

I have a lot of questions.. can you clarify:

• What are your incoming Windows Logs so we can understand if these are generic OTS Windows logs or if they are syslog in nature?
• How these incoming Windows Logs sitting on the HF you mentioned in your title or from other UFs maybe?
• Are these 2 F5 VIPs doubling as indexers for your Splunk architecture or do these incoming logs get to indexers at some point?

[tiaatim]

They are wineventlog - app/sec/sys

They are coming in to the HF via UF.

The data gets sent to an index cluster via another app/stanza on the HF and they look fine in Splunk.

Thanks!

[13tsavage]

Are you sending these as [syslog] to the 2 F5 VIPs because that is the only port open on those hosts?
Is the indexer cluster stanza using [syslog] and has to send via port 514(UDP) too?

[tiaatim]

No, the indexers are receiving the data via standard Splunk indexer port 9997. I'm sending via syslog to the F5 because thats the way the host behind the VIPs wants to see it.