Getting Data In

How to route and filter data on the Heavy Forwarder to separate indexer groups?

mookiie2005
Communicator

We need to route and filter data on the heavy forwarder. We are having trouble configuring the routing of security logs to a Splunk instance specifically for security logs and the main Enterprise instance.

We want to direct certain logfiles to our main indexers and/or a separate Splunk instance specifically for security. We want to send security data to the security instance and send windows application/system logs to both sets of indexers. We created an app on the heavy forwarder, however, it does not seem to be working as expected.

Here is our props.conf:

[WinEventLog:Application]
TRANSFORMS-routing_Windows_=Windows_GIS_data_app

[WinEventLog:Security]
TRANSFORMS-routing_Windows_=Windows_GIS_data_sec

[WinEventLog:System]
TRANSFORMS-routing_Windows_=Windows_GIS_data_sys

Main index

[Perfmon:CPU Load]
TRANSFORMS-routing_Windows_=Windows_splunk_main_data

[Perfmon:Available Memory]
TRANSFORMS-routing_Windows_=Windows_splunk_main_data

[Perfmon:Free Disk Space]
TRANSFORMS-routing_Windows_=Windows_splunk_main_data

Perfmon index

[Perfmon:PhysicalDisk]
TRANSFORMS-routing_Windows_=Windows_splunk_perfmon_data

[Perfmon:CPU]
TRANSFORMS-routing_Windows_=Windows_splunk_perfmon_data

[Perfmon:Memory]
TRANSFORMS-routing_Windows_=Windows_splunk_perfmon_data

[Perfmon:MemoryStats]
TRANSFORMS-routing_Windows_=Windows_splunk_perfmon_data

[Perfmon:CPUTime]
TRANSFORMS-routing_Windows_=Windows_splunk_perfmon_data

[Perfmon:FreeDiskSpace]
TRANSFORMS-routing_Windows_=Windows_splunk_perfmon_data

Here is our transforms.conf:

[Windows_GIS_data_app]
REGEX = .
DEST_KEY = _TCP_ROUTING
FORMAT = ALL_INDEXERS

[Windows_GIS_data_sec]
REGEX = .
DEST_KEY = _TCP_ROUTING
FORMAT = GIS_INDEXERS

[Windows_GIS_data_sys]
REGEX = .
DEST_KEY = _TCP_ROUTING
FORMAT = ALL_INDEXERS

[Windows_splunk_main_data]
REGEX = .
DEST_KEY = _TCP_ROUTING
FORMAT = FARMERS_MAIN_INDEXERS

[Windows_splunk_perfmon_data]
REGEX = .
DEST_KEY = _TCP_ROUTING
FORMAT = FARMERS_INDEXERS

Here is our outputs.conf:

[indexAndForward]
index=true
selectiveIndexing=true 


[GIS_INDEXERS]
indexAndForward = true

[tcpout:GIS_INDEXERS]
server=10.148.186.83:9997, 10.148.186.84:9997



[ALL_INDEXERS]
indexAndForward = true

[tcpout:ALL_INDEXERS]
server=10.142.114.13:18017, 10.148.186.83:9997, 10.148.186.84:9997


[FARMERS_INDEXERS]
indexAndForward = true

[tcpout:FARMERS_INDEXERS]
server=10.142.114.13:18015

[FARMERS_MAIN_INDEXERS]
indexAndForward = false

[tcpout:FARMERS_MAIN_INDEXERS]
server=10.142.114.13:18013

Can anyone help to resolve the issue?

1 Solution

sk314
Builder

The main trick here is that you can specify multiple tcpout routes/groups in transforms.conf FORMAT line.

So to clear things up with your example (assuming gis is security only)

outputs.conf
[tcpout:farmers_perfmon]
server=10.142.114.13:18015

[tcpout:farmers_main]
server=10.142.114.13:18013

[tcpout:security_only]
server=10.148.186.83:9997, 10.148.186.84:9997

[tcpout:general_indexer]
server=10.142.114.13:18017

transforms.conf

[Windows_GIS_data_app]
REGEX = .
DEST_KEY = _TCP_ROUTING
FORMAT = general_indexer,security_only

[Windows_GIS_data_sec]
REGEX = .
DEST_KEY = _TCP_ROUTING
FORMAT = security_only

[Windows_GIS_data_sys]
REGEX = .
DEST_KEY = _TCP_ROUTING
FORMAT = general_indexer,security_only

[Windows_splunk_main_data]
REGEX = .
DEST_KEY = _TCP_ROUTING
FORMAT = farmers_main

[Windows_splunk_perfmon_data]
REGEX = .
DEST_KEY = _TCP_ROUTING
FORMAT = farmers_perfmon

AFAIK - the only difference from your solution is the ALL_INDEXERS tcpout group load balances to the three servers - which is not what you want. You want two copies - one copy to go to 10.142.114.13:18017 and the other copy to go to (10.148.186.83:9997, 10.148.186.84:9997). This is fixed in my approach by having two TCP routing groups in transforms.

Hope this helps.

View solution in original post

aaraneta_splunk
Splunk Employee
Splunk Employee

@mookiie2005 - Did one of the answers below help provide a solution your question? If yes, please click “Accept” below the best answer to resolve this post. If no, please leave a comment with more feedback. Thanks.

0 Karma

sk314
Builder

Did it help?

0 Karma

sk314
Builder

The main trick here is that you can specify multiple tcpout routes/groups in transforms.conf FORMAT line.

So to clear things up with your example (assuming gis is security only)

outputs.conf
[tcpout:farmers_perfmon]
server=10.142.114.13:18015

[tcpout:farmers_main]
server=10.142.114.13:18013

[tcpout:security_only]
server=10.148.186.83:9997, 10.148.186.84:9997

[tcpout:general_indexer]
server=10.142.114.13:18017

transforms.conf

[Windows_GIS_data_app]
REGEX = .
DEST_KEY = _TCP_ROUTING
FORMAT = general_indexer,security_only

[Windows_GIS_data_sec]
REGEX = .
DEST_KEY = _TCP_ROUTING
FORMAT = security_only

[Windows_GIS_data_sys]
REGEX = .
DEST_KEY = _TCP_ROUTING
FORMAT = general_indexer,security_only

[Windows_splunk_main_data]
REGEX = .
DEST_KEY = _TCP_ROUTING
FORMAT = farmers_main

[Windows_splunk_perfmon_data]
REGEX = .
DEST_KEY = _TCP_ROUTING
FORMAT = farmers_perfmon

AFAIK - the only difference from your solution is the ALL_INDEXERS tcpout group load balances to the three servers - which is not what you want. You want two copies - one copy to go to 10.142.114.13:18017 and the other copy to go to (10.148.186.83:9997, 10.148.186.84:9997). This is fixed in my approach by having two TCP routing groups in transforms.

Hope this helps.

View solution in original post

mookiie2005
Communicator

This is very good information. It turned out that for some reason the Heavy Forwarders we were trying to send the data to the separate instance was actually refusing the data on port 9997. We changed this to 9996 and opened that port on the HF and that resolved the issue. This was particularly frustrating as we had established communication between our indexers and the HF of the other instance in an earlier test, but settings were changed on the HF which cause this issue for us.

0 Karma

DarthDMader
Explorer

Hi,

In your description is the inputs missing:
You are searching for '_TCP_ROUTING ='
See also:
the docs for outputs.conf (the last example) and the inputs.conf

Kind Regards
Darth

0 Karma
Did you miss .conf21 Virtual?

Good news! The event's keynotes and many of its breakout sessions are now available online, and still totally FREE!