Send data to heavy forwarder to filter events AND change sourcetype - help please

[johnmvang]

Hello,

As the question states, i'm looking to send events from a universal forwarder to a heavy forwarder to have filtered. Once filtered, i'd like to change the sourcetype. I have not implemented this yet. This is for me to propose to upper management to agree on. I want to make sure the props/transforms piece is correct. I think the filtering is good, however i just want to make sure the syntax is all good.

I've listed my config and config details:

ON UNIVERSAL FORWARDER

inputs.conf

[monitor://c:\program files\app1\web.log]
_TCP_ROUTING = filter_heavy_forwarders
index = cmis_index

sourcetype = app1_web_logs

ON UNIVERSAL FORWARDER

outputs.conf

[tcpout]
defaultGroup=infosec_indexers

[tcpout:infosec_indexers]
autoLB = true
server = infosec_server1:9997,infosec_server2:9997,infosec_server3:9997…,infosec_server16:9997

[tcpout:cmis_indexers]
autoLB = true
server = cmis_server1:9997

[tcpout:filter_heavy_forwarders]
autoLB = true

Server = filter_hvyfwd1:9998,filter_hvyfwd2:9998

ON HEAVY FORWARDER

props.conf

[app1_web_logs]
TRANSFORMS-routing = app1_web_filter

TRANSFORMS-changest = app1_cmis_web

ON HEAVY FORWARDER

transforms.conf

[app1_web_filter]
REGEX = (Events|To|Filter)
DEST_KEY = _TCP_ROUTING
FORMAT = cmis_indexers

[app1_cmis_web_st]
DEST_KEY = MetaData:Sourcetype

FORMAT = sourcetype::app1_cmis_web

ON HEAVY FORWARDER

outputs.conf

[tcpout]
defaultGroup=none

[tcpout:cmis_indexers]
autoLB = true

server = cmis_server1:9997

[gcusello]

Hi johnmvang,
only just a few information:
in UFs outputs.conf I don't see

[tcpout-server://infosec_server1:9997]
[tcpout-server://infosec_server2:9997]
[tcpout-server://infosec_server3:9997]
[tcpout-server://infosec_server16:9997]

but probably you missed these rows only in the question.

On HF, you send all transforming logs only to cmis_indexers?
if yes you don't need in props.conf TRANSFORMS-routing = app1_web_filter and the relative stanza in transforms.conf.
In addition I suggest to perform selective addressing directly in UFs.

Anyway, I think that the problem is in HFs transforms.conf: the REGEX row is missing, so add REGEX = . to the [app1_cmis_web_st] stanza.

Bye.
Giuseppe

[gjanders]

You can use btool to validate your syntax.
I notice you don't mention both tcpout's within the outputs.conf but this might be from the universal forwarder only.

[johnmvang]

i updated my question with the unifwd outputs. But let me look into the btool and i'll come back.

Thanks,

John