How do I exclude everything but certain events at my heavy forwarder?

[dewald13]

At my HF I want to exclude everything BUT three websites. I have been playing with this for days now, that's what she said, with no luck. Below is what I have in transforms.conf and props.conf

Transforms.conf

[setnull]

REGEX = .

DEST_KEY = queue

FORMAT = nullQueue

[keep1]

REGEX = \[google.com\]

DEST_KEY = queue

FORMAT = indexQueue

[keep2]

REGEX = \[nascar.com\]

DEST_KEY = queue

FORMAT = indexQueue

[keep3]

REGEX = \[cnn.com\]

DEST_KEY = queue

FORMAT = indexQueue

Props.conf

[source::tcp:9999]

TRANSFORMS-set= setnull,keep1,keep2,keep3

With these settings, we are still indexing ALL websites coming in on that port including the three listed. We are trying to ONLY keep google, nascar and cnn.

[sdaniels]

If for some reason you've got a Universal forwarder for Light weight forwarder this wouldn't work. But on the HF this looks good.

[glitchcowboy]

I set this up on a new full instance of splunk and it works, given this input from some client:

 # echo "[cnn.com]" > /dev/tcp/splunkindexerip/9999
 # echo "[http://hi.net]" > /dev/tcp/splunkindexerip/9999
 # echo "[google.com]" > /dev/tcp/splunkindexerip/9999
 # echo "[foxnews.com]" > /dev/tcp/splunkindexerip/9999
 # echo "[startitup.com]" > /dev/tcp/splunkindexerip/9999
 # echo "[samsonite.com]" > /dev/tcp/splunkindexerip/9999
 # echo "[nascar.com]" > /dev/tcp/splunkindexerip/9999

props.conf

[source::tcp:5555]
TRANSFORMS-set = setnull,keep,keep2,keep3

transforms.conf

[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[keep]
REGEX = \[cnn.com\]
DEST_KEY = queue
FORMAT = indexQueue

[keep2]
REGEX = \[google.com\]
DEST_KEY = queue
FORMAT = indexQueue

[keep3]
REGEX = \[nascar.com\]
DEST_KEY = queue
FORMAT = indexQueue

The output was what you were hoping for, only cnn, google, and nascar show up.

Is the Heavy Forwarder actually indexing before it passes the data on?

[Ayn]

OK, so do you have event samples so we can see what you're matching against?

[dewald13]

I tried it with JUST the setnull transform and that is working. That stopped all data coming in from the port.

[Ayn]

It looks correct.

Also, do you have any sample events so we can see what exactly you're matching against?

[dewald13]

I will try that but before I do does my Source in Props.conf look correct? I would hate to test this and not have it apply only to the port we are working with.

[Ayn]

Did you try this with JUST the setnull transform to check that your props.conf settings are actually applied?

[dewald13]

There are no quotatino marks in my config. I should not have put those in my reply. The forwardslash is not showing up in my posts for some reason.

Any other ideas?

[Ayn]

Putting quotation marks in there will make the regex engine actually look for quotation marks, not escape your whole string. You need to escape the brackets like sdaniels shows in his example.

[dewald13]

Is my spacing correct?

[dewald13]

I do have the "\" in all of the REGEX to escape the brackets. For some reason it did not show up in this post. I have also tried it with no brackets "REGEX= nascar" and that did not work either....

[sdaniels]

Are you trying to match the square brackets on [nascar] etc? You'll need to escape the bracket it so it will look like this:

REGEX=\[nascar\]

Right now I don't think your regex is working.