topic Send data to heavy forwarder to filter events AND change sourcetype - help please in Getting Data In

Send data to heavy forwarder to filter events AND change sourcetype - help please

johnmvang — Tue, 29 Sep 2020 16:10:01 GMT

Hello,

As the question states, i'm looking to send events from a universal forwarder to a heavy forwarder to have filtered. Once filtered, i'd like to change the sourcetype. I have not implemented this yet. This is for me to propose to upper management to agree on. I want to make sure the props/transforms piece is correct. I think the filtering is good, however i just want to make sure the syntax is all good.

I've listed my config and config details:

ON UNIVERSAL FORWARDER

inputs.conf

[monitor://c:\program files\app1\web.log]
_TCP_ROUTING = filter_heavy_forwarders
index = cmis_index

sourcetype = app1_web_logs

ON UNIVERSAL FORWARDER

outputs.conf

[tcpout]
defaultGroup=infosec_indexers

[tcpout:infosec_indexers]
autoLB = true
server = infosec_server1:9997,infosec_server2:9997,infosec_server3:9997…,infosec_server16:9997

[tcpout:cmis_indexers]
autoLB = true
server = cmis_server1:9997

[tcpout:filter_heavy_forwarders]
autoLB = true

Server = filter_hvyfwd1:9998,filter_hvyfwd2:9998

ON HEAVY FORWARDER

props.conf

[app1_web_logs]
TRANSFORMS-routing = app1_web_filter

TRANSFORMS-changest = app1_cmis_web

ON HEAVY FORWARDER

transforms.conf

[app1_web_filter]
REGEX = (Events|To|Filter)
DEST_KEY = _TCP_ROUTING
FORMAT = cmis_indexers

[app1_cmis_web_st]
DEST_KEY = MetaData:Sourcetype

FORMAT = sourcetype::app1_cmis_web

ON HEAVY FORWARDER

outputs.conf

[tcpout]
defaultGroup=none

[tcpout:cmis_indexers]
autoLB = true

server = cmis_server1:9997

Re: Send data to heavy forwarder to filter events AND change sourcetype - help please

gjanders — Fri, 13 Oct 2017 01:02:33 GMT

You can use btool to validate your syntax.
I notice you don't mention both tcpout's within the outputs.conf but this might be from the universal forwarder only.

Re: Send data to heavy forwarder to filter events AND change sourcetype - help please

johnmvang — Fri, 13 Oct 2017 02:56:44 GMT

i updated my question with the unifwd outputs. But let me look into the btool and i'll come back.

Thanks,

John

Re: Send data to heavy forwarder to filter events AND change sourcetype - help please

gcusello — Tue, 29 Sep 2020 16:14:43 GMT

Hi johnmvang,
only just a few information:
in UFs outputs.conf I don't see

[tcpout-server://infosec_server1:9997]
[tcpout-server://infosec_server2:9997]
[tcpout-server://infosec_server3:9997]
[tcpout-server://infosec_server16:9997]

but probably you missed these rows only in the question.

On HF, you send all transforming logs only to cmis_indexers?
if yes you don't need in props.conf TRANSFORMS-routing = app1_web_filter and the relative stanza in transforms.conf.
In addition I suggest to perform selective addressing directly in UFs.

Anyway, I think that the problem is in HFs transforms.conf: the REGEX row is missing, so add REGEX = . to the [app1_cmis_web_st] stanza.

Bye.
Giuseppe