topic Re: How to optimize my search to find 30+ src_ips and any dest_ip that are not 2 specific IP addresses? in Splunk Search

How to optimize my search to find 30+ src_ips and any dest_ip that are not 2 specific IP addresses?

packet_hunter — Tue, 29 Sep 2020 10:42:36 GMT

Currently I am using (OR)s

For example:

Index = A sourcetype=a (src="192.168.3.5" OR src="192.168.3.6" OR.... etc.) | stats list(dest_ip) by src_ip

Currently I filter thru the dest_ip list, but I would like to add a NOT dest_ip= 10.10.3.5 OR NOT dest_ip=10.10.3.6

Any advice to optimize and speed up the search is greatly appreciated!

Thank you

Re: How to optimize my search to find 30+ src_ips and any dest_ip that are not 2 specific IP addresses?

sundareshr — Fri, 19 Aug 2016 22:07:40 GMT

See if this works (haven't tried it)

index=a sourcetype=b NOT (dest_ip=10.10.3.5 OR dest_ip=10.10.3.6) | regex src="([0-9]{2})$"

Re: How to optimize my search to find 30+ src_ips and any dest_ip that are not 2 specific IP addresses?

packet_hunter — Fri, 19 Aug 2016 23:21:38 GMT

Do you have any suggestions for speeding up a search with >30 IPs: e.g. src="" OR src="" etc

I have been considering the use of a lookup table (with all the IPs), but not sure if that will speed the search up.

I am searching thru firewall logs and my time frame is last 30 days, but it always times out when I use all the OR(s).

Thank you

Re: How to optimize my search to find 30+ src_ips and any dest_ip that are not 2 specific IP addresses?

sundareshr — Sat, 20 Aug 2016 00:34:38 GMT

Summary Index is probably your best bet.

http://docs.splunk.com/Documentation/Splunk/6.4.2/Knowledge/Usesummaryindexing

Re: How to optimize my search to find 30+ src_ips and any dest_ip that are not 2 specific IP addresses?

packet_hunter — Sat, 20 Aug 2016 00:40:43 GMT

the IP lists change often, does an inputlookup speed things up?

Re: How to optimize my search to find 30+ src_ips and any dest_ip that are not 2 specific IP addresses?

sundareshr — Sat, 20 Aug 2016 00:52:07 GMT

inputlookup will not only just create the same "OR" search, it will include a sub-search (extra step). Don't see that being faster.

Look at "Common causes of slow searches" section

http://wiki.splunk.com/Community:PerformanceTroubleshooting

Re: How to optimize my search to find 30+ src_ips and any dest_ip that are not 2 specific IP addresses?

packet_hunter — Sat, 20 Aug 2016 00:54:39 GMT

Ok, I follow what you are saying, thank you

Re: How to optimize my search to find 30+ src_ips and any dest_ip that are not 2 specific IP addresses?

packet_hunter — Mon, 22 Aug 2016 13:22:17 GMT

I have been looking into Summary indexing. Please make summary index your answer and I will accept.

Thank you

Re: How to optimize my search to find 30+ src_ips and any dest_ip that are not 2 specific IP addresses?

stevejfice — Tue, 23 Aug 2016 15:43:25 GMT

It might depend on how your data is being indexed but with my Palo Alto data I can do the following:

index=pan src_ip<192.168.2.40 src_ip>192.168.2.20 | stats list(dest_ip) by src_ip

(obviously my source ip's are different but the syntax works).

Interestingly if I switch the two src_ip searches around (greater than first) it fails to work, not sure why but I'm sure a more intelligent person than me can find out)

Re: How to optimize my search to find 30+ src_ips and any dest_ip that are not 2 specific IP addresses?

packet_hunter — Wed, 24 Aug 2016 15:05:26 GMT

Thank you for the reply and interesting observation.
My problem is that I often need to search a number of internet src IPs that are unrelated but I will try ordering them and see it that does anything. My primary effort is too speed up the search process without summary indexing b/c the IPs of concern change frequently.

Thank you again for your reply, I will let you know the results of my testing if there is anything identifiable.