Getting Data In

Send data to heavy forwarder to filter events AND change sourcetype - help please

johnmvang
Path Finder

Hello,

As the question states, i'm looking to send events from a universal forwarder to a heavy forwarder to have filtered. Once filtered, i'd like to change the sourcetype. I have not implemented this yet. This is for me to propose to upper management to agree on. I want to make sure the props/transforms piece is correct. I think the filtering is good, however i just want to make sure the syntax is all good.

I've listed my config and config details:

ON UNIVERSAL FORWARDER

inputs.conf

[monitor://c:\program files\app1\web.log]
_TCP_ROUTING = filter_heavy_forwarders
index = cmis_index

sourcetype = app1_web_logs

ON UNIVERSAL FORWARDER

outputs.conf

[tcpout]
defaultGroup=infosec_indexers

[tcpout:infosec_indexers]
autoLB = true
server = infosec_server1:9997,infosec_server2:9997,infosec_server3:9997…,infosec_server16:9997

[tcpout:cmis_indexers]
autoLB = true
server = cmis_server1:9997

[tcpout:filter_heavy_forwarders]
autoLB = true

Server = filter_hvyfwd1:9998,filter_hvyfwd2:9998

ON HEAVY FORWARDER

props.conf

[app1_web_logs]
TRANSFORMS-routing = app1_web_filter

TRANSFORMS-changest = app1_cmis_web

ON HEAVY FORWARDER

transforms.conf

[app1_web_filter]
REGEX = (Events|To|Filter)
DEST_KEY = _TCP_ROUTING
FORMAT = cmis_indexers

[app1_cmis_web_st]
DEST_KEY = MetaData:Sourcetype

FORMAT = sourcetype::app1_cmis_web

ON HEAVY FORWARDER

outputs.conf

[tcpout]
defaultGroup=none

[tcpout:cmis_indexers]
autoLB = true

server = cmis_server1:9997

0 Karma

gcusello
Legend

Hi johnmvang,
only just a few information:
in UFs outputs.conf I don't see

[tcpout-server://infosec_server1:9997]
[tcpout-server://infosec_server2:9997]
[tcpout-server://infosec_server3:9997]
[tcpout-server://infosec_server16:9997]

but probably you missed these rows only in the question.

On HF, you send all transforming logs only to cmis_indexers?
if yes you don't need in props.conf TRANSFORMS-routing = app1_web_filter and the relative stanza in transforms.conf.
In addition I suggest to perform selective addressing directly in UFs.

Anyway, I think that the problem is in HFs transforms.conf: the REGEX row is missing, so add REGEX = . to the [app1_cmis_web_st] stanza.

Bye.
Giuseppe

0 Karma

gjanders
SplunkTrust
SplunkTrust

You can use btool to validate your syntax.
I notice you don't mention both tcpout's within the outputs.conf but this might be from the universal forwarder only.

0 Karma

johnmvang
Path Finder

i updated my question with the unifwd outputs. But let me look into the btool and i'll come back.

Thanks,

John

0 Karma
Get Updates on the Splunk Community!

This Week's Community Digest - Splunk Community Happenings [9.26.22]

Get the latest news and updates from the Splunk Community here! Upcoming User Group Events! 👏 Check ...

BSides Splunk 2022 - The Call for Papers is now Open!

TLDR; Main Site: https://bsidessplunk.com CFP Site: https://bsidessplunk.com/cfp CFP Opens: December 15th, ...

Sending Metrics to Splunk Enterprise With the OpenTelemetry Collector

This blog post is part of an ongoing series on OpenTelemetry. The OpenTelemetry project is the second largest ...