Getting Data In

Filter events before indexing

rbw78
Communicator

Hello,

I'm consulting the documentation regarding filtering events before they get indexed but i have issue to understand how i could do that.

I got events coming from 1 IP, and i don't want to index the events where the field "risk_rating" is lower than 75.

i really don't know how to do that with props.conf and transforms.conf

Props.conf
[host::10.6.75.16]
TRANSFORMS-null= setnull

Tranforms.conf
[setnull]
REGEX=???? /did i have to use this field/
DEST_KEY=queue
FORMAT=nullQueue

How could i say all the values under 75 for the field "risk_rating" isn't indexed under the transforms file ?
It seems quit simple but i really don't get it ...

Thanks.

0 Karma

Ayn
Legend

It's not really simple unfortunately because the filter strings that are used for event filtering can't be used for mathematical comparisons like that. The only way to filter events is by constructing a regular expression that matches your unwanted events - or vice versa, a regular expression that "whitelists" events that should not be filtered.

If you have something like the text "risk_rating=50" or similar and you want to grab only events where this rating is below 75, it CAN be done with a regex - it doesn't look very pretty but it works.

REGEX = risk_rating=([0-6]|7[0-4])

richard_g_curry
Explorer

Ayn's regex will work if you add the quotes to it, like this:

REGEX = risk_rating=\"([0-6]|7[0-4])\"
0 Karma

Ayn
Legend

What does the regex you're using now look like? My sample regex was assuming that the equals sign was directly followed by the digits, so if you've used mine straight away that won't work - you'll have to add the quotes too.

0 Karma

rbw78
Communicator

Thanks for the feedback Ayn.

Well, i tried the REGEX you told me but it dosen't seems to work, i still have risk_rating events lower than 75 indexed.

This field look like this in my events

risk_rating="00"

I've only changed the REGEX value into the transforms.conf mentioned above.
It should be enough right ?

0 Karma

Ayn
Legend

The regex I showed you does this. There's no need to create 75 different matching groups, because just two will do: one for when risk_rating starts with anything from 0 to 6, and one for when it starts with 7 and is followed by anything from 0 to 4.

There's a theoretical performance impact, but in fairness there's lots of stuff going on when indexing data so I think that impact is neglectable.

0 Karma

rbw78
Communicator

Ok thanks for the feedback.

Well, the possible values i want to exclude for this field goes from 0 to 74.
If i add a regex for this 75 possibilities, is there a performance impact for indexing the coming data ?

0 Karma
Get Updates on the Splunk Community!

Splunk Observability Cloud | Unified Identity - Now Available for Existing Splunk ...

Raise your hand if you’ve already forgotten your username or password when logging into an account. (We can’t ...

Index This | How many sides does a circle have?

February 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

Registration for Splunk University is Now Open!

Are you ready for an adventure in learning?   Brace yourselves because Splunk University is back, and it's ...