syslog (udp:514) filtering

[weigeltf]

Hi,

I am wondering if my question couldn't get answered quickly. I have parsed many very similiar questions and tried to find my way out from the answers. Here is my situation.

I get too many syslog messages on sourec::udp:514 as sourcetype=syslog. Within those messages many are not of interest for my purpose. So I want to get rid of some of them to protect my indexing license against overflow.

As I said, I have done some configuration but I am absolutely unsure if this works and if, how I can control and check the behaviour. Is there somebody out in the field who is willing to answer my questions.

Thanks for your help.

props.conf

[sourcetype::syslog]

TRANSFORMS-asa= asa_teardown_null,asa_built_null

transforms.conf

[asa_teardown_null]

DEST_KEY = queue

FORMAT = nullQueue

REGEX = (ASA-6-302014|ASA-6-302016|ASA-6-302018|ASA-6-302021|ASA-6-302304|ASA-6
-305010|ASA-6-305012|ASA-6-603109|ASA-6-617100)

[asa_built_null]

DEST_KEY = queue

FORMAT = nullQueue

REGEX = (ASA-6-302003|ASA-6-302009|ASA-6-302013|ASA-6-302017|ASA-6-302020|ASA-6
-305009|ASA-6-305011|ASA-6-603108|ASA-6-302013|ASA-6-302015)

[jbsplunk · Accepted solution]

[sourcetype::syslog] is not a valid way of calling a props.conf entry for sourcetype. The rest of the transform looks ok. However, this syntax needs to be corrected or things won't work. If 'syslog' is the sourcetype of these events, your stanza should look like this:

props.conf

[syslog]

TRANSFORMS-asa= asa_teardown_null,asa_built_null

props.conf.spec can be found in $SPLUNK_HOME/etc/system/README, or @ http://docs.splunk.com/Documentation/Splunk/latest/admin/propsconf

Please see the following section:

[<spec>]
* This stanza enables properties for a given <spec>.
* A props.conf file can contain multiple stanzas for any number of different <spec>.
* Follow this stanza name with any number of the following attribute/value pairs, as appropriate
  for what you want to do.
* If you do not set an attribute for a given <spec>, the default is used.

<spec> can be:
1. <sourcetype>, the source type of an event.
2. host::<host>, where <host> is the host for an event.
3. source::<source>, where <source> is the source for an event.
4. rule::<rulename>, where <rulename> is a unique name of a source type classification rule.
5. delayedrule::<rulename>, where <rulename> is a unique name of a delayed source type
   classification rule.
These are only considered as a last resort before generating a new source type based on the
source seen.

Please note that there isn't a sourcetype:: specification anywhere in this area.

[jbsplunk]

You can't see the specific events that were routed to nullQueue, which is kind of the point. However, you will see entries in metrics.log:

INFO Metrics - group=pipeline, name=dev-null, processor=nullqueue, cpu_seconds=0.000000, executes=4450, cumulative_hits=3469242

[weigeltf]

jbsplunk, thanks.
Good and fast answer. It works. I have checked and the nullQueue regexed records don't appear any longer.

[weigeltf]

sourcetype=syslog comes out of the facts that I deciced to takes this as the sourcetype. No rewrite takes place. I used the stanza [source::udp:8514] which raieses the same questions.

By the way, syslog udp port 514 is natted to udp port 8514 with iptables. That works quite fine.

inputs.conf
[udp://8514]
sourcetype = syslog

[Ayn]

Do you know for a fact that your data has sourcetype "syslog" when Splunk indexes it, or is it dynamically rewritten in a transform somewhere? Because if it is, that could cause the stanza in props.conf not to take effect.