Getting Data In

Syslog-ng filter filter props/transforms for data routing

Splunk_citizen
Explorer

Hello Splunkers,

Earlier we were using central syslog-ng server to capture all /var/log/messages from hosts now we have installed UF on unix servers and we are going with UF monitoring instead of syslog route
We dont want capture all events in /var/log/messages, below ones is previous filters in syslog-ng.conf files in host.
So now my question is how i can write a transform/props to reflects same like below syslog-ng.conf file and deploy locally in forwards i order to capture /var/log/messages to monitor.

# destinations
destination d_sshpam { file("/var/adm/syslog/sshd.log"); };
destination d_sshpamb { file("/var/adm/syslog/sshd.log"); };
#destination d_sapdelete { file("/dev/null"); };
#destination d_sapruntime { file("/dev/null"); };
#destination d_emfbackground { file("/dev/null"); };
destination d_messages { file("/var/adm/syslog/syslog.log"); };

destination d_logserver {syslog("xx.xx.xx.xx" transport(tcp)); };

log { source(s_local); filter(f_sshpam); destination(d_sshpam); flags(final); };

log { source(s_local); filter(f_sshpamb); destination(d_sshpamb); flags(final); };

#log { source(s_local); filter(f_sapdelete); destination(d_sapdelete); flags(final); };
#
#log { source(s_local); filter(f_sapruntime); destination(d_sapruntime); flags(final); };

#log { source(s_local); filter(f_emfbackground); destination(d_emfbackground); flags(final); };

log {
source(s_local);

destination(d_messages);
destination(d_logserver);
};
0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi Splunk_citizen,
you can filter data on Indexers or on Heavy Forwarders.
To filter events you can see at http://docs.splunk.com/Documentation/Splunk/7.1.2/Forwarding/Routeandfilterdatad
In easy words, you have to:

  • identify regexes of the logs you want to index or to discard,
  • indert in props.con the following row in the sourcetype of your syslog:

    [my_sourcetype]
    TRANSFORMS-filter = setnull,setparsing

  • insert in transforms.conf the following rows (beware that the stanza's names are the same of TRANSFORMS command:

    [setnull]
    REGEX = .
    DEST_KEY = queue
    FORMAT = nullQueue
    [setparsing]
    REGEX = my_regex
    DEST_KEY = queue
    FORMAT = indexQueue

Bye.
Giuseppe

0 Karma

Splunk_citizen
Explorer

Thank you Casello,

That is the syslog file and not sure what kind of regex i have to build.

0 Karma
Got questions? Get answers!

Join the Splunk Community Slack to learn, troubleshoot, and make connections with fellow Splunk practitioners in real time!

Meet up IRL or virtually!

Join Splunk User Groups to connect and learn in-person by region or remotely by topic or industry.

Get Updates on the Splunk Community!

Build the Future of Agentic AI: Join the Splunk Agentic Ops Hackathon

AI is changing how teams investigate incidents, detect threats, automate workflows, and build intelligent ...

[Puzzles] Solve, Learn, Repeat: Character substitutions with Regular Expressions

This challenge was first posted on Slack #puzzles channelFor BORE at .conf23, we had a puzzle question which ...

Splunk Community Badges!

  Hey everyone! Ready to earn some serious bragging rights in the community? Along with our existing badges ...