How to optimize my search to find 30+ src_ips and any dest_ip that are not 2 specific IP addresses?

[packet_hunter]

Currently I am using (OR)s

For example:

Index = A sourcetype=a (src="192.168.3.5" OR src="192.168.3.6" OR.... etc.) | stats list(dest_ip) by src_ip

Currently I filter thru the dest_ip list, but I would like to add a NOT dest_ip= 10.10.3.5 OR NOT dest_ip=10.10.3.6

Any advice to optimize and speed up the search is greatly appreciated!

Thank you

[stevejfice]

It might depend on how your data is being indexed but with my Palo Alto data I can do the following:

index=pan src_ip<192.168.2.40 src_ip>192.168.2.20 | stats list(dest_ip) by src_ip

(obviously my source ip's are different but the syntax works).

Interestingly if I switch the two src_ip searches around (greater than first) it fails to work, not sure why but I'm sure a more intelligent person than me can find out)

[packet_hunter]

Thank you for the reply and interesting observation.
My problem is that I often need to search a number of internet src IPs that are unrelated but I will try ordering them and see it that does anything. My primary effort is too speed up the search process without summary indexing b/c the IPs of concern change frequently.

Thank you again for your reply, I will let you know the results of my testing if there is anything identifiable.

[sundareshr · Accepted solution]

Summary Index is probably your best bet.

http://docs.splunk.com/Documentation/Splunk/6.4.2/Knowledge/Usesummaryindexing

[packet_hunter]

the IP lists change often, does an inputlookup speed things up?

[sundareshr]

inputlookup will not only just create the same "OR" search, it will include a sub-search (extra step). Don't see that being faster.

Look at "Common causes of slow searches" section

http://wiki.splunk.com/Community:PerformanceTroubleshooting

[packet_hunter]

I have been looking into Summary indexing. Please make summary index your answer and I will accept.

Thank you

[packet_hunter]

Ok, I follow what you are saying, thank you

[sundareshr]

See if this works (haven't tried it)

index=a sourcetype=b NOT (dest_ip=10.10.3.5 OR dest_ip=10.10.3.6) | regex src="([0-9]{2})$"

[packet_hunter]

Do you have any suggestions for speeding up a search with >30 IPs: e.g. src="" OR src="" etc

I have been considering the use of a lookup table (with all the IPs), but not sure if that will speed the search up.

I am searching thru firewall logs and my time frame is last 30 days, but it always times out when I use all the OR(s).

Thank you