I am trying to build a report for AWS FlowLogs which can be used to analyze SG usage. Specifically, I want a list of incoming traffic (by 'dest_ip') which shows all IP/port combinations. Unfortunatel...
See more...
I am trying to build a report for AWS FlowLogs which can be used to analyze SG usage. Specifically, I want a list of incoming traffic (by 'dest_ip') which shows all IP/port combinations. Unfortunately, a simple 'stats count by dest_ip,dest_port,protocol,src_ip,src_port' does not result in a usable report -- because all the stateful return-traffic is listed, too. There are 10K's worth of incoming packets with dest_port in the 1024-65535 range, i.e., where that particular 'dest' server had initiated a connection using an ephemeral local port and then the return traffic went to the same port. So 99% of the 'incoming' ports are not actual listeners which we need to include in our SGs.
I have spent hours testing various combinations of filters, e.g. count<5, or dest_port>18000 or (dest_port>1024 AND src_port<1024) or even a 'where NOT IN(src_port,22,53,80,3389, etc)'. But we have a lot of services which use high-port numbers so all these methods accidentally remove valid traffic.
Instead, I think the only accurate method would be one where each connection is evaluated for:
- is the incoming 'dest_port' above 1024?
- if so, is there a corresponding packet in the preceding 1000 ms, i.e., identical-but-reversed dest and src IP/ports?
- if so, assume this later packet is the return from a stateful request sent on an ephemeral port -- remove it from the results!
Has anyone else run into this situation, and what was your solution? Thank you for any suggestions!