Getting Data In

Heavy Forwarder Load Balancing syslog data to F5 VIPs

tiaatim
Path Finder

We are on 7.2.5.1. My outputs is sending incoming Windows logs out to 2 F5 VIPs via a syslog stanza. The data is going out and only ever hits the first vip in the server= line in the stanza

[syslog:test_group]
priority = NO_PRI
server = 10.X.X.1:514,10.X.X.2:514
type = udp

The .1 is receiving all of the data on the F5 and the HF never seems to switch over to the .2 IP.

Any help would be greatly appreciated.

0 Karma

codebuilder
SplunkTrust
SplunkTrust

Your configuration is not correct for what you're trying to accomplish.

The "server" parameter in the [syslog] stanza takes only a single IP:PORT and is where you define the address to your syslog server (singular).
This parameter is not for load balancing, and is the reason only one of your two defined IP's is receiving data.

https://docs.splunk.com/Documentation/Splunk/8.0.1/Admin/Outputsconf#Syslog_output----

----
An upvote would be appreciated and Accept Solution if it helps!
0 Karma

tiaatim
Path Finder

Thanks, so do something like this?

[syslog]
defaultGroup = syslog:test_group1, syslog:test_group2
type = udp
priority = NO_PRI

[syslog:test_group1]
server = 10.X.X.1:514

[syslog:test_group2]
server = 10.X.X.2:514

0 Karma

codebuilder
SplunkTrust
SplunkTrust

It's a bit of a one-off configuration, and I'm not sure your results will be consistent, but this config combo should work. Though I can't be certain you won't get duplicate data.

[syslog]
defaultGroup = syslog:test_group1, syslog:test_group2
type = udp
priority = NO_PRI

[syslog:test_group1]
server = 10.X.X.1:514

[syslog:test_group2]
server = 10.X.X.2:514

----
An upvote would be appreciated and Accept Solution if it helps!
0 Karma

tiaatim
Path Finder

I tried the above config and unfortunately it didn't work. It still only sent to one IP address. Here is the outputs.conf that I tried:

[syslog]
defaultGroup = syslog:test_group1,syslog:test_group2
type = udp
priority = NO_PRI

[syslog:test_group1]
server = 10.x.x.1:514

[syslog:test_group2]
server = 10.x.x.2:514

0 Karma

codebuilder
SplunkTrust
SplunkTrust

A number of things could cause that. Did you cycle Splunk after making the config change? Is the F5 using a wildcard on the IP? Have you tried commenting out test_group1 to verify data reaches test_group2?

Again, what you're trying to do is a one-off, and isn't really meant to be configured this way. It may honestly be time to re-think your approach on this.

----
An upvote would be appreciated and Accept Solution if it helps!
0 Karma

tiaatim
Path Finder

Yes, I did restart Splunk. I didn't comment out test_group1 this time but did before I made this specific change and saw data going to group2. Thanks for your help.

0 Karma

codebuilder
SplunkTrust
SplunkTrust

You could try the example in the documentation, which will work, and also ensure you don't get duplicate data.

#
# The following example shows how to route events to syslog server
# This is similar to tcpout routing, but DEST_KEY is set to _SYSLOG_ROUTING
#

# 1. Edit $SPLUNK_HOME/etc/system/local/props.conf and set a TRANSFORMS-routing
#    attribute:
[default]
TRANSFORMS-routing=errorRouting

[syslog]
TRANSFORMS-routing=syslogRouting

# 2. Edit $SPLUNK_HOME/etc/system/local/transforms.conf and set errorRouting
#    and syslogRouting rules:
[errorRouting]
REGEX=error
DEST_KEY=_SYSLOG_ROUTING
FORMAT=errorGroup

[syslogRouting]
REGEX=.
DEST_KEY=_SYSLOG_ROUTING
FORMAT=syslogGroup

# 3. Edit $SPLUNK_HOME/etc/system/local/outputs.conf and set which syslog
#    outputs go to with servers or groups:
[syslog]
defaultGroup=everythingElseGroup

[syslog:syslogGroup]
server = 10.1.1.197:9997

[syslog:errorGroup]
server=10.1.1.200:9999

[syslog:everythingElseGroup]
server=10.1.1.250:6666

https://docs.splunk.com/Documentation/Splunk/latest/Admin/Outputsconf?utm_source=answers&utm_medium=...

----
An upvote would be appreciated and Accept Solution if it helps!
0 Karma

codebuilder
SplunkTrust
SplunkTrust

I'm assuming that you are using two F5's for redundancy, but you can configure a single VIP across two clustered F5's (much easier in haproxy), and that would solve your issue.

----
An upvote would be appreciated and Accept Solution if it helps!
0 Karma

tiaatim
Path Finder

Currently the 2 VIPs are on one F5. We had one VIP going to a 2 IP pool but saw that traffic was only going to a single IP in the pool. So we are now trying 2 VIPs with the 2 IPs in the pool's flipped so the first VIP goes to .3 and .4 in the pool and the second VIP goes to .4 and .3 in the pool. Make sense?

0 Karma

13tsavage
Communicator

I have a lot of questions.. can you clarify:

  • What are your incoming Windows Logs so we can understand if these are generic OTS Windows logs or if they are syslog in nature?
  • How these incoming Windows Logs sitting on the HF you mentioned in your title or from other UFs maybe?
  • Are these 2 F5 VIPs doubling as indexers for your Splunk architecture or do these incoming logs get to indexers at some point?
0 Karma

tiaatim
Path Finder

They are wineventlog - app/sec/sys

They are coming in to the HF via UF.

The data gets sent to an index cluster via another app/stanza on the HF and they look fine in Splunk.

Thanks!

0 Karma

13tsavage
Communicator

Are you sending these as [syslog] to the 2 F5 VIPs because that is the only port open on those hosts?
Is the indexer cluster stanza using [syslog] and has to send via port 514(UDP) too?

0 Karma

tiaatim
Path Finder

No, the indexers are receiving the data via standard Splunk indexer port 9997. I'm sending via syslog to the F5 because thats the way the host behind the VIPs wants to see it.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...