Splunk search for identifying the docker container connections to unusual ports?

[dinesh2]

I am looking for the search query which can give me a result of any docker container connections to unusal ports. 
Tired the below query 

index=aws_eks_* responseObject.spec.limits{}.type=*container* | NOT search port IN (80,443,8080,8443,3000,330)=80 OR port=443 OR port=8080 OR port=8443 OR port=3000 OR port=3306)

[PickleRick]

OK. There are several things wrong with this search.

1. There is no command "NOT search". If you wanted to do it that way, you should do
| search NOT (your set of conditions).

2. This is a very, very ineffective way of searching. Doing a big search, possibly returning a huge number of events only to narrow that down further down the pipeline is wasting server's resources. In some cases splunk is able to optimize it out on its own but in general all conditions should be included in the base search in order to limit returned events as much as possible.

3. "port IN (80,443,8080,8443,3000,330)=80" - what is this supposed to mean? It doesn't make sense you copy-pasted it from somewhere? Because the "port IN (80,443,8080,8443,3000,330)" is indeed equivalent of "port=80 OR port=443 OR port=8080 OR port=8443 OR port=3000 OR port=3306"?

4. After you "fix" the points 2 and 3, you get something like this:

index=aws_eks_* responseObject.spec.limits{}.type=*container* NOT (port IN (80,443,8080,8443,3000,330))

It's still far from perfect though.

You should _never_ (at least not in the base search) use wildcards at the beginning of your sought for term. If you look for "*container*", splunk has to read _all_ events from the given time range (or at least all those to which the rest of the condition can limit your result set) to verify whether anywhere within the message there is such a string which has your "container" word in the middle. Since you used wildcard at the beginning of the search term, splunk cannot just use its list of indexed terms. It has to read every raw event. That's very very ineffective. So if you have some predefined list of those types, or at least you know that all of them have values starting with the same or just a few prefixes, do it like

responseObject.spec.limits{}.type IN (my_container*,your_container*,whatever_container*)

It will be much faster.

5. Are you sure you have a field just caled "port" in your data? The other field you're searching on seems to be some deeper-nested json field so are you sure the field is just called "port" and not - for example - responseObject.port or something like that?

6. In general, inclusion is faster than exclusion so if you can list specific ports you're looking for it would be faster than telling "find me something not matching my criteria". But that's something you can't always avoid so treat it more like a general remark.

7. Be aware that negation can work in two different ways. A (NOT port=something) condition will match any event which does not have a field called "port" having value of "something". Which can also include events not having e field called "port" at all. But if you write the condition as (port!=something), it will match only those events which do have the field called "port" but for which said field contains value other than "something". So the difference is that events without that port would be included in one search but wouldn't be included in the other.

8. This kind of search (see point 6) might be quite slow due to how splunk works so it could be very useful to use some form of acceleration (the proper solution here would depend on several factors so can't give you a ready-made answer which is best).