Splunk Search

How do I exclude everything but certain events at my heavy forwarder?

dewald13
Path Finder

At my HF I want to exclude everything BUT three websites. I have been playing with this for days now, that's what she said, with no luck. Below is what I have in transforms.conf and props.conf

Transforms.conf

[setnull]

REGEX = .

DEST_KEY = queue

FORMAT = nullQueue

[keep1]

REGEX = \[google.com\]

DEST_KEY = queue

FORMAT = indexQueue

[keep2]

REGEX = \[nascar.com\]

DEST_KEY = queue

FORMAT = indexQueue

[keep3]

REGEX = \[cnn.com\]

DEST_KEY = queue

FORMAT = indexQueue

Props.conf

[source::tcp:9999]

TRANSFORMS-set= setnull,keep1,keep2,keep3

With these settings, we are still indexing ALL websites coming in on that port including the three listed. We are trying to ONLY keep google, nascar and cnn.

sdaniels
Splunk Employee
Splunk Employee

If for some reason you've got a Universal forwarder for Light weight forwarder this wouldn't work. But on the HF this looks good.

0 Karma

glitchcowboy
Path Finder

I set this up on a new full instance of splunk and it works, given this input from some client:

 # echo "[cnn.com]" > /dev/tcp/splunkindexerip/9999
 # echo "[http://hi.net]" > /dev/tcp/splunkindexerip/9999
 # echo "[google.com]" > /dev/tcp/splunkindexerip/9999
 # echo "[foxnews.com]" > /dev/tcp/splunkindexerip/9999
 # echo "[startitup.com]" > /dev/tcp/splunkindexerip/9999
 # echo "[samsonite.com]" > /dev/tcp/splunkindexerip/9999
 # echo "[nascar.com]" > /dev/tcp/splunkindexerip/9999

props.conf

[source::tcp:5555]
TRANSFORMS-set = setnull,keep,keep2,keep3

transforms.conf

[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[keep]
REGEX = \[cnn.com\]
DEST_KEY = queue
FORMAT = indexQueue

[keep2]
REGEX = \[google.com\]
DEST_KEY = queue
FORMAT = indexQueue

[keep3]
REGEX = \[nascar.com\]
DEST_KEY = queue
FORMAT = indexQueue

The output was what you were hoping for, only cnn, google, and nascar show up.

Is the Heavy Forwarder actually indexing before it passes the data on?

Ayn
Legend

OK, so do you have event samples so we can see what you're matching against?

0 Karma

dewald13
Path Finder

I tried it with JUST the setnull transform and that is working. That stopped all data coming in from the port.

Ayn
Legend

It looks correct.

Also, do you have any sample events so we can see what exactly you're matching against?

0 Karma

dewald13
Path Finder

I will try that but before I do does my Source in Props.conf look correct? I would hate to test this and not have it apply only to the port we are working with.

Ayn
Legend

Did you try this with JUST the setnull transform to check that your props.conf settings are actually applied?

0 Karma

dewald13
Path Finder

There are no quotatino marks in my config. I should not have put those in my reply. The forwardslash is not showing up in my posts for some reason.

Any other ideas?

0 Karma

Ayn
Legend

Putting quotation marks in there will make the regex engine actually look for quotation marks, not escape your whole string. You need to escape the brackets like sdaniels shows in his example.

0 Karma

dewald13
Path Finder

Is my spacing correct?

0 Karma

dewald13
Path Finder

I do have the "\" in all of the REGEX to escape the brackets. For some reason it did not show up in this post. I have also tried it with no brackets "REGEX= nascar" and that did not work either....

0 Karma

sdaniels
Splunk Employee
Splunk Employee

Are you trying to match the square brackets on [nascar] etc? You'll need to escape the bracket it so it will look like this:

REGEX=\[nascar\]

Right now I don't think your regex is working.

0 Karma
Get Updates on the Splunk Community!

Federated Search for Amazon S3 | Key Use Cases to Streamline Compliance Workflows

Modern business operations are supported by data compliance. As regulations evolve, organizations must ...

New Dates, New City: Save the Date for .conf25!

Wake up, babe! New .conf25 dates AND location just dropped!! That's right, this year, .conf25 is taking place ...

Introduction to Splunk Observability Cloud - Building a Resilient Hybrid Cloud

Introduction to Splunk Observability Cloud - Building a Resilient Hybrid Cloud  In today’s fast-paced digital ...