Getting Data In

syslog (udp:514) filtering

weigeltf
New Member

Hi,

I am wondering if my question couldn't get answered quickly. I have parsed many very similiar questions and tried to find my way out from the answers. Here is my situation.

I get too many syslog messages on sourec::udp:514 as sourcetype=syslog. Within those messages many are not of interest for my purpose. So I want to get rid of some of them to protect my indexing license against overflow.

As I said, I have done some configuration but I am absolutely unsure if this works and if, how I can control and check the behaviour. Is there somebody out in the field who is willing to answer my questions.

Thanks for your help.

props.conf

[sourcetype::syslog]

TRANSFORMS-asa= asa_teardown_null,asa_built_null

transforms.conf

[asa_teardown_null]

DEST_KEY = queue

FORMAT = nullQueue

REGEX = (ASA-6-302014|ASA-6-302016|ASA-6-302018|ASA-6-302021|ASA-6-302304|ASA-6
-305010|ASA-6-305012|ASA-6-603109|ASA-6-617100)

[asa_built_null]

DEST_KEY = queue

FORMAT = nullQueue

REGEX = (ASA-6-302003|ASA-6-302009|ASA-6-302013|ASA-6-302017|ASA-6-302020|ASA-6
-305009|ASA-6-305011|ASA-6-603108|ASA-6-302013|ASA-6-302015)

0 Karma
1 Solution

jbsplunk
Splunk Employee
Splunk Employee

[sourcetype::syslog] is not a valid way of calling a props.conf entry for sourcetype. The rest of the transform looks ok. However, this syntax needs to be corrected or things won't work. If 'syslog' is the sourcetype of these events, your stanza should look like this:

props.conf

[syslog]

TRANSFORMS-asa= asa_teardown_null,asa_built_null

props.conf.spec can be found in $SPLUNK_HOME/etc/system/README, or @ http://docs.splunk.com/Documentation/Splunk/latest/admin/propsconf

Please see the following section:

[<spec>]
* This stanza enables properties for a given <spec>.
* A props.conf file can contain multiple stanzas for any number of different <spec>.
* Follow this stanza name with any number of the following attribute/value pairs, as appropriate
  for what you want to do.
* If you do not set an attribute for a given <spec>, the default is used.

<spec> can be:
1. <sourcetype>, the source type of an event.
2. host::<host>, where <host> is the host for an event.
3. source::<source>, where <source> is the source for an event.
4. rule::<rulename>, where <rulename> is a unique name of a source type classification rule.
5. delayedrule::<rulename>, where <rulename> is a unique name of a delayed source type
   classification rule.
These are only considered as a last resort before generating a new source type based on the
source seen.

Please note that there isn't a sourcetype:: specification anywhere in this area.

View solution in original post

jbsplunk
Splunk Employee
Splunk Employee

[sourcetype::syslog] is not a valid way of calling a props.conf entry for sourcetype. The rest of the transform looks ok. However, this syntax needs to be corrected or things won't work. If 'syslog' is the sourcetype of these events, your stanza should look like this:

props.conf

[syslog]

TRANSFORMS-asa= asa_teardown_null,asa_built_null

props.conf.spec can be found in $SPLUNK_HOME/etc/system/README, or @ http://docs.splunk.com/Documentation/Splunk/latest/admin/propsconf

Please see the following section:

[<spec>]
* This stanza enables properties for a given <spec>.
* A props.conf file can contain multiple stanzas for any number of different <spec>.
* Follow this stanza name with any number of the following attribute/value pairs, as appropriate
  for what you want to do.
* If you do not set an attribute for a given <spec>, the default is used.

<spec> can be:
1. <sourcetype>, the source type of an event.
2. host::<host>, where <host> is the host for an event.
3. source::<source>, where <source> is the source for an event.
4. rule::<rulename>, where <rulename> is a unique name of a source type classification rule.
5. delayedrule::<rulename>, where <rulename> is a unique name of a delayed source type
   classification rule.
These are only considered as a last resort before generating a new source type based on the
source seen.

Please note that there isn't a sourcetype:: specification anywhere in this area.

jbsplunk
Splunk Employee
Splunk Employee

You can't see the specific events that were routed to nullQueue, which is kind of the point. However, you will see entries in metrics.log:

INFO Metrics - group=pipeline, name=dev-null, processor=nullqueue, cpu_seconds=0.000000, executes=4450, cumulative_hits=3469242

0 Karma

weigeltf
New Member

jbsplunk, thanks.
Good and fast answer. It works. I have checked and the nullQueue regexed records don't appear any longer.

0 Karma

weigeltf
New Member

sourcetype=syslog comes out of the facts that I deciced to takes this as the sourcetype. No rewrite takes place. I used the stanza [source::udp:8514] which raieses the same questions.

By the way, syslog udp port 514 is natted to udp port 8514 with iptables. That works quite fine.

inputs.conf
[udp://8514]
sourcetype = syslog

0 Karma

Ayn
Legend

Do you know for a fact that your data has sourcetype "syslog" when Splunk indexes it, or is it dynamically rewritten in a transform somewhere? Because if it is, that could cause the stanza in props.conf not to take effect.

0 Karma
Get Updates on the Splunk Community!

Last Chance to Submit Your Paper For BSides Splunk - Deadline is August 12th!

Hello everyone! Don't wait to submit - The deadline is August 12th! We have truly missed the community so ...

Ready, Set, SOAR: How Utility Apps Can Up Level Your Playbooks!

 WATCH NOW Powering your capabilities has never been so easy with ready-made Splunk® SOAR Utility Apps. Parse ...

DevSecOps: Why You Should Care and How To Get Started

 WATCH NOW In this Tech Talk we will talk about what people mean by DevSecOps and deep dive into the different ...