I want to black hole 'last message repeated xxx times' from my syslog output BEFORE it gets to the indexer

[mwk]

This message is fairly repetitive (10's of thousands a month) in one of our systems and it's just taking up space in splunk.

I've spent the better part of the last two days trying to get this to work. Using a heavy forwarder on the syslog server and trying to get it to "Discard specific events and keep the rest"
Route and filter data

I've got a props.conf on the heavy forwarder

[source::/var/log/rsyslog/iswb1/syslog]
TRANSFORMS-null= setnull

and a transforms.conf on the heavy forwarder

[setnull]
REGEX = \[sshd\]
DEST_KEY = queue
FORMAT = nullQueue

in /opt/splunk/etc/system/local. I even tried to just emulate the example to see if all the parts are connected to each other and working correctly. I used logger to send 'sshd' through the from the system that rsyslog is monitoring and that dutifully ends up in my splunk indexer. Obviously doing something wrong, probably a forest for the trees problem at this point.

[mwk · Accepted solution]

So I seem to have gotten it all to play nice. props.conf talking to transforms.conf correctly now and here's the regex that's killing my annoying messages (at least it seems to be working. I've had this issue for so long I'll believe it when I don't see them for a day or two.)

in the transforms.conf this syntax
REGEX = (?s)(last).+?(message).+?(repeated.).+(times)

seems to be the winner.

[lguinn2]

What is in your outputs.conf on the heavy forwarder?

[mwk]

outputs.conf

[tcpout]
defaultGroup = default-autolb-group

[tcpout:default-autolb-group]
server = indexer.server.name:9997

[tcpout-server://indexer.server.name:9997]

FWIW the transforms.conf and props.conf seem to be working now. I can get it to stop specific one word messages that I hand feed the rsyslog server from the monitored server but now of course I'm stuck in regex hell trying to get the correct combination to have it get rid of:

2014-12-30T14:12:03-06:00 last message repeated 447 times

Currently trying this but it doesn't seem to be working.

[setnull-server]
REGEX = (?s)(last).+?(message).+?(repeated.)
DEST_KEY = queue
FORMAT = nullQueue

[ppablo]

Hi @mwk

Have you considered using a blacklist rule to filter out this unwanted data? Check out the documentation here.
http://docs.splunk.com/Documentation/Splunk/6.2.1/Data/Whitelistorblacklistspecificincomingdata#Blac...

[mwk]

Blacklists are on a file level right? I need everything else that's being logged to the file. Just want some part of splunk to filter out the noise. FWIW rsyslog has trouble getting rid of this as well or it wouldn't even be something would see.

2014-12-30T12:11:16-06:00 last message repeated 3 times
2014-12-30T12:12:01-06:00 last message repeated 3 times
2014-12-30T12:12:53-06:00 last message repeated 3 times
2014-12-30T12:13:44-06:00 last message repeated 11 times
2014-12-30T12:14:29-06:00 last message repeated 6 times