Getting Data In

Heavy Forwarder routing to two seperate indexer clusters

fisuser1
Contributor

I have a HF forwarding specific sourcetypes to two different indexer clusters. However, it does not seem to be working even though I have followed routing and filter data approach. All data seems to be forwarding to all indexers. (seeing errors on the second set of indexers since the index does not exist, which is expected) I've also confirmed there isn't any conflicting inputs taking precedence in system/local.

https://docs.splunk.com/Documentation/Splunk/7.3.1/Forwarding/Routeandfilterdatad

Any suggestions?

outputs
[tcpout]
indexAndForward = 0

[tcpout]
defaultGroup = fis_search_peers,profile_search_peers

[tcpout:fis_search_peers]
server = idx1:9997, idx2:9997, idx3:9997, idx4:9997, idx5:9997, idx6:9997, idx7:9997, idx8:9997
autoLBFrequency=15
autoLB=true
useACK=true
forceTimebasedAutoLB = true

[tcpout:profile_search_peers]
server = f_idx1:9997, f_idx2:9997, f_idx3:9997
autoLBFrequency=15
autoLB=true
useACK=true
forceTimebasedAutoLB = true

inputs
[integrated_payables]
connection = prd_integrated_payables
disabled = 0
index = integrated_payables
index_time_mode = dbColumn
input_timestamp_column_number = 2
interval = 900
mode = rising
query = SELECT [Id] ,\
[Date] ,\
[Thread] ,\
[Level] ,\
Logger ,\
NestedContext ,\
[Message] ,\
[Exception] ,\
[HostName] ,\
[Thread_Context_ID]\
FROM "Logging"."dbo"."Log"\
WHERE Date > ?\
ORDER BY Date ASC
sourcetype = prd_int_pay_payspan_logging_db
tail_rising_column_number = 2
_TCP_ROUTING = fis_search_peers

[monitor:///ma/xpressng/logs/WAS85//MA_XpressNG_Prd_ma.log]
sourcetype = prd_mba_infinity_application
index = mba_infinity
_TCP_ROUTING = profile_search_peers

0 Karma
1 Solution

renjith_nair
SplunkTrust
SplunkTrust

@fisuser1 ,

You have set defaultGroup to both indexer groups and as per documentation, it sends events to all specified target group

https://docs.splunk.com/Documentation/Forwarder/7.3.1/Forwarder/Configureforwardingwithoutputs.conf#...

Set default target groups in outputs.conf

The defaultGroup attribute lets you set default groups for automatic forwarding at the global level, in your [tcpout] stanza.

The defaultGroup specifies one or more target groups that you define later in tcpout:<target_group> stanzas. The forwarder sends all events to the specified groups.    
[tcpout]
defaultGroup= <target_group1>, <target_group2>, ...
If you do not want to forward data automatically, do not set the defaultGroup attribute.
Happy Splunking!

View solution in original post

renjith_nair
SplunkTrust
SplunkTrust

@fisuser1 ,

You have set defaultGroup to both indexer groups and as per documentation, it sends events to all specified target group

https://docs.splunk.com/Documentation/Forwarder/7.3.1/Forwarder/Configureforwardingwithoutputs.conf#...

Set default target groups in outputs.conf

The defaultGroup attribute lets you set default groups for automatic forwarding at the global level, in your [tcpout] stanza.

The defaultGroup specifies one or more target groups that you define later in tcpout:<target_group> stanzas. The forwarder sends all events to the specified groups.    
[tcpout]
defaultGroup= <target_group1>, <target_group2>, ...
If you do not want to forward data automatically, do not set the defaultGroup attribute.
Happy Splunking!

fisuser1
Contributor

somehow I completely overlooked this, thank you @renjith.nair, I believe this will do exactly what I am looking for in breaking up this traffic. Thank you!

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...