How to group search results for large amount of data?

[tmuhieddine]

Hello,

First all, forgive me I am new at using Splunk, hoping someone can help me out. We use our SPLUNK instance to log Firewall traffic, specifically Cisco ASA FWs. We currently have 1 firewall that has a wide open (any to any) FW rule in place. We are looking to build rules and eventually remove that wide open rule.

We searched for traffic for the last 30 day's hitting that rule and it returned millions of events. Obviously most of these are duplicate sessions, my question is that I am looking for a way to group the results per host or modify the search to only return single hosts and not duplicate entry's. I will need to export this data and build some FW policy's, obviously with the number of events this would not be ideal.

Current Search string:
eventtype=cisco-firewall host="TC-CADC-SFW-5203" action="" src_ip="" dest_ip="" src_port="" dest_port="" transport="" service="*" 0x7b3c7918 (This Hex number represents the wide open rule)

Example of event returned:
Oct 17 2019 10:04:38 TC-CADC-SFW-5203 : %ASA-5-106100: access-list global_access permitted tcp corp/10.16.46.197(50575) -> WAN/10.48.16.176(9100) hit-cnt 1 first hit [0x7b3c7918, 0x0]
host = TC-CADC-SFW-5203source = udp:8514sourcetype = cisco:asa

All suggestions welcome!

Thanks

[richgalloway]

There are a few ways to group events. The cluster command automatically groups like events. dedup removes events with exact duplicates of a given field or fields. You also use stats to group events by one or more fields.

eventtype=cisco-firewall host="TC-CADC-SFW-5203" action="" src_ip="" dest_ip="" src_port="" dest_port="" transport="" service="" 0x7b3c7918 | cluster

eventtype=cisco-firewall host="TC-CADC-SFW-5203" action="" src_ip="" dest_ip="" src_port="" dest_port="" transport="" service="" 0x7b3c7918 | dedup _raw

eventtype=cisco-firewall host="TC-CADC-SFW-5203" action="" src_ip="" dest_ip="" src_port="" dest_port="" transport="" service="" 0x7b3c7918 | stats values(*) as * by src_ip

[tmuhieddine]

Thank you! I will give this a shot and let you know how it goes!

Much appreciated!!