topic Re: Syslog-ng filter filter props/transforms for data routing in Getting Data In

Syslog-ng filter filter props/transforms for data routing

Splunk_citizen — Sun, 09 Sep 2018 19:04:02 GMT

Hello Splunkers,

Earlier we were using central syslog-ng server to capture all /var/log/messages from hosts now we have installed UF on unix servers and we are going with UF monitoring instead of syslog route
We dont want capture all events in /var/log/messages, below ones is previous filters in syslog-ng.conf files in host.
So now my question is how i can write a transform/props to reflects same like below syslog-ng.conf file and deploy locally in forwards i order to capture /var/log/messages to monitor.

# destinations
destination d_sshpam { file("/var/adm/syslog/sshd.log"); };
destination d_sshpamb { file("/var/adm/syslog/sshd.log"); };
#destination d_sapdelete { file("/dev/null"); };
#destination d_sapruntime { file("/dev/null"); };
#destination d_emfbackground { file("/dev/null"); };
destination d_messages { file("/var/adm/syslog/syslog.log"); };

destination d_logserver {syslog("xx.xx.xx.xx" transport(tcp)); };

log { source(s_local); filter(f_sshpam); destination(d_sshpam); flags(final); };

log { source(s_local); filter(f_sshpamb); destination(d_sshpamb); flags(final); };

#log { source(s_local); filter(f_sapdelete); destination(d_sapdelete); flags(final); };
#
#log { source(s_local); filter(f_sapruntime); destination(d_sapruntime); flags(final); };

#log { source(s_local); filter(f_emfbackground); destination(d_emfbackground); flags(final); };

log {
source(s_local);

destination(d_messages);
destination(d_logserver);
};

Re: Syslog-ng filter filter props/transforms for data routing

gcusello — Tue, 29 Sep 2020 21:10:16 GMT

Hi Splunk_citizen,
you can filter data on Indexers or on Heavy Forwarders.
To filter events you can see at http://docs.splunk.com/Documentation/Splunk/7.1.2/Forwarding/Routeandfilterdatad
In easy words, you have to:

identify regexes of the logs you want to index or to discard,
indert in props.con the following row in the sourcetype of your syslog:

[my_sourcetype]
TRANSFORMS-filter = setnull,setparsing
insert in transforms.conf the following rows (beware that the stanza's names are the same of TRANSFORMS command:

[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue
[setparsing]
REGEX = my_regex
DEST_KEY = queue
FORMAT = indexQueue

Bye.
Giuseppe

Re: Syslog-ng filter filter props/transforms for data routing

Splunk_citizen — Mon, 10 Sep 2018 17:28:23 GMT

Thank you Casello,

That is the syslog file and not sure what kind of regex i have to build.