Getting Data In

Syslog-ng filter filter props/transforms for data routing

Splunk_citizen
Explorer

Hello Splunkers,

Earlier we were using central syslog-ng server to capture all /var/log/messages from hosts now we have installed UF on unix servers and we are going with UF monitoring instead of syslog route
We dont want capture all events in /var/log/messages, below ones is previous filters in syslog-ng.conf files in host.
So now my question is how i can write a transform/props to reflects same like below syslog-ng.conf file and deploy locally in forwards i order to capture /var/log/messages to monitor.

# destinations
destination d_sshpam { file("/var/adm/syslog/sshd.log"); };
destination d_sshpamb { file("/var/adm/syslog/sshd.log"); };
#destination d_sapdelete { file("/dev/null"); };
#destination d_sapruntime { file("/dev/null"); };
#destination d_emfbackground { file("/dev/null"); };
destination d_messages { file("/var/adm/syslog/syslog.log"); };

destination d_logserver {syslog("xx.xx.xx.xx" transport(tcp)); };

log { source(s_local); filter(f_sshpam); destination(d_sshpam); flags(final); };

log { source(s_local); filter(f_sshpamb); destination(d_sshpamb); flags(final); };

#log { source(s_local); filter(f_sapdelete); destination(d_sapdelete); flags(final); };
#
#log { source(s_local); filter(f_sapruntime); destination(d_sapruntime); flags(final); };

#log { source(s_local); filter(f_emfbackground); destination(d_emfbackground); flags(final); };

log {
source(s_local);

destination(d_messages);
destination(d_logserver);
};
0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi Splunk_citizen,
you can filter data on Indexers or on Heavy Forwarders.
To filter events you can see at http://docs.splunk.com/Documentation/Splunk/7.1.2/Forwarding/Routeandfilterdatad
In easy words, you have to:

  • identify regexes of the logs you want to index or to discard,
  • indert in props.con the following row in the sourcetype of your syslog:

    [my_sourcetype]
    TRANSFORMS-filter = setnull,setparsing

  • insert in transforms.conf the following rows (beware that the stanza's names are the same of TRANSFORMS command:

    [setnull]
    REGEX = .
    DEST_KEY = queue
    FORMAT = nullQueue
    [setparsing]
    REGEX = my_regex
    DEST_KEY = queue
    FORMAT = indexQueue

Bye.
Giuseppe

0 Karma

Splunk_citizen
Explorer

Thank you Casello,

That is the syslog file and not sure what kind of regex i have to build.

0 Karma
Get Updates on the Splunk Community!

Extending Observability Content to Splunk Cloud

Watch Now!   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to leverage ...

More Control Over Your Monitoring Costs with Archived Metrics!

What if there was a way you could keep all the metrics data you need while saving on storage costs?This is now ...

New in Observability Cloud - Explicit Bucket Histograms

Splunk introduces native support for histograms as a metric data type within Observability Cloud with Explicit ...