topic Re: Filter private ip from src AND dest in Splunk Search

Filter private ip from src AND dest

thin_air — Tue, 25 Jan 2022 22:10:55 GMT

New to the community so all help is appreciated!

Requirement
We have a requirement to filter some network data in a correlation search to return any data which has a public ip in the "src" or "dest" field.

Solution
I tried several variants of this:
... | search (src!="10.0.0.0/8" OR src!="192.168.0.0/16" OR src!="172.16.0.0/12") AND (dest!="10.0.0.0/8" OR dest!="192.168.0.0/16" OR dest!="172.16.0.0/12")

I boiled it down to this, which also does not work:
... | search src!="10.0.0.0/8" AND dest!="10.0.0.0/8"

It appears that my query is evaluating the first "OR" individually, meaning that no matter what I set the dest!= filter to it does not return results.

My Request
Clearly I don't understand the logic being used for OR/AND operators and a better understanding of that would be appreciated. Ultimately though, I'm not stuck on this logic, so if there is a better way to only return results which has a public ip in the src OR dest fields I'm happy to learn the best way to do that as well!

Thanks in advance for the help!

Re: Filter private ip from src AND dest

ITWhisperer — Tue, 25 Jan 2022 23:02:30 GMT

Your logic has been inverted by the use of !=

Consider this simple case

X="A"

if (X != "A" OR X != "B")

Since X = "A" it fails the first condition but is true for the second condition

What you should have is

if (X != "A" AND X != "B")

So, in your case it should be more like

... | search (src!="10.0.0.0/8" AND src!="192.168.0.0/16" AND src!="172.16.0.0/12") OR (dest!="10.0.0.0/8" AND dest!="192.168.0.0/16" AND dest!="172.16.0.0/12")

Re: Filter private ip from src AND dest

johnhuang — Wed, 26 Jan 2022 02:15:46 GMT

The solution provided by @ITWhisperer works.

Another solution which works significantly faster (3x) in my testing is to use cidrmatch.

Try this:

base search... | eval src_net=IF(cidrmatch("10.0.0.0/8", src) OR cidrmatch("172.16.0.0/12", src) OR cidrmatch("192.168.0.0/16", src), "private", "public") | eval dest_net=IF(cidrmatch("10.0.0.0/8", dest) OR cidrmatch("172.16.0.0/12", dest) OR cidrmatch("192.168.0.0/16", dest), "private", "public") | search src_net="public" OR dest_net="public"

Keep in mind that the eval to determine the network type is not necessary, but gives you some flexibility

Re: Filter private ip from src AND dest

ITWhisperer — Wed, 26 Jan 2022 06:35:43 GMT

The solution given by @johnhuang does not work if the values of src and dest are actually in CIDR notation e.g. "10.0.0.0/8" (as seemed to be implied by the question), however, it does work if these fields actually contain ip addresses e.g. anything in the range 10.x.x.x, which, to be fair, is possibly/probably the case, although since no example events were included, I, for one, have no idea. 😀

The point I was making was more to do with how the logic of using not equals inverts the use of OR and AND. This applies to other logical expressions, not just ip address and/or cidr notation, which is why I used a simple example to explain the point.

Re: Filter private ip from src AND dest

johnhuang — Wed, 26 Jan 2022 06:55:53 GMT

Excuses. I did acknowlege that your solution works and gave your a karma. What more do you want?

In all seriousness, traffic logs are usually huge datasources and it's worth the effort to optimize. Some of my major pain points has been this.

Re: Filter private ip from src AND dest

thin_air — Wed, 26 Jan 2022 15:02:39 GMT

Appreciate both responses and I understand more about why my original attempts failed.

I went with @johnhuang solution because it offers more flexibility but the solution by @ITWhisperer is a wonderful demonstration of why my attempts failed and I learned a lot from it.

Thank you both!! 😄