I used @jconger's article Splunking Microsoft Azure Network Watcher Data to configure the Splunk Add-on for Microsoft Cloud Services to ingest Azure Network Security Group (NSG) flow logs. @jconger also provided a useful props.conf to clean up the JSON blobs that come in.
However, the JSON objects contain one or more tuples (connection flows), which result in having multiple src_ips, dest_ip, etc. in a single event. I need to separate each tuple into its own event so I can map the events into the Network Traffic data model. One thought is to use SEDCMD to remove everything but the tuples; this has two caveats: (1) I'm not sure how to create an effective RegEx for this, and (2) I lose fields such as "rule" and "resourceID" (contains NSG name) that apply to all tuples in a single JSON object. Here are some sample logs:
{"time":"2019-01-09T23:14:42.9159948Z","systemId":"4ed3b8cc-3bd0-4e6f-ae79-ffeea35a858d","category":"NetworkSecurityGroupFlowEvent","resourceId":"/SUBSCRIPTIONS/2E6A794A-489A-4824-AB75-DCDC4D0ECDC5/RESOURCEGROUPS/CENTRAL_RG/PROVIDERS/MICROSOFT.NETWORK/NETWORKSECURITYGROUPS/NSG-POC-PUBLIC","operationName":"NetworkSecurityGroupFlowEvents","properties":{"Version":1,"flows":[{"rule":"UserRule_DefaultInboundDenyAll","flows":[{"mac":"000D3A97A1BA","flowTuples":["1547075623,104.248.168.1,172.20.0.132,51353,8088,T,I,D"]}]},{"rule":"UserRule_AllowInternetOut","flows":[{"mac":"000D3A97A1BA","flowTuples":["1547075626,172.20.0.132,52.239.177.196,14690,443,T,O,A","1547075626,172.20.0.132,52.239.177.196,14691,443,T,O,A","1547075626,172.20.0.132,52.239.177.196,14692,443,T,O,A","1547075628,172.20.0.132,208.91.113.71,123,123,U,O,A","1547075628,172.20.0.132,208.91.112.51,123,123,U,O,A","1547075628,172.20.0.132,208.91.112.50,123,123,U,O,A","1547075634,172.20.0.132,208.91.113.70,123,123,U,O,A","1547075652,172.20.0.132,52.239.177.196,14694,443,T,O,A","1547075652,172.20.0.132,52.239.177.196,14695,443,T,O,A","1547075652,172.20.0.132,52.239.177.196,14696,443,T,O,A","1547075677,172.20.0.132,52.239.177.196,14698,443,T,O,A","1547075677,172.20.0.132,52.239.177.196,14699,443,T,O,A","1547075677,172.20.0.132,52.239.177.196,14700,443,T,O,A"]}]}]}}
{"time":"2019-01-09T23:13:42.9136121Z","systemId":"4ed3b8cc-3bd0-4e6f-ae79-ffeea35a858d","category":"NetworkSecurityGroupFlowEvent","resourceId":"/SUBSCRIPTIONS/2E6A794A-489A-4824-AB75-DCDC4D0ECDC5/RESOURCEGROUPS/CENTRAL_RG/PROVIDERS/MICROSOFT.NETWORK/NETWORKSECURITYGROUPS/NSG-POC-PUBLIC","operationName":"NetworkSecurityGroupFlowEvents","properties":{"Version":1,"flows":[{"rule":"UserRule_DefaultInboundDenyAll","flows":[{"mac":"000D3A97A1BA","flowTuples":["1547075589,194.28.115.245,172.20.0.132,47661,33897,T,I,D","1547075614,178.128.44.249,172.20.0.132,59511,8088,T,I,D"]}]},{"rule":"UserRule_AllowInternetOut","flows":[{"mac":"000D3A97A1BA","flowTuples":["1547075575,172.20.0.132,52.239.177.196,14682,443,T,O,A","1547075575,172.20.0.132,52.239.177.196,14683,443,T,O,A","1547075575,172.20.0.132,52.239.177.196,14684,443,T,O,A","1547075601,172.20.0.132,52.239.177.196,14686,443,T,O,A","1547075601,172.20.0.132,52.239.177.196,14687,443,T,O,A","1547075601,172.20.0.132,52.239.177.196,14688,443,T,O,A"]}]}]}}
{"time":"2019-01-09T23:12:42.9104712Z","systemId":"4ed3b8cc-3bd0-4e6f-ae79-ffeea35a858d","category":"NetworkSecurityGroupFlowEvent","resourceId":"/SUBSCRIPTIONS/2E6A794A-489A-4824-AB75-DCDC4D0ECDC5/RESOURCEGROUPS/CENTRAL_RG/PROVIDERS/MICROSOFT.NETWORK/NETWORKSECURITYGROUPS/NSG-POC-PUBLIC","operationName":"NetworkSecurityGroupFlowEvents","properties":{"Version":1,"flows":[{"rule":"UserRule_DefaultInboundDenyAll","flows":[{"mac":"000D3A97A1BA","flowTuples":["1547075521,165.76.160.44,172.20.0.132,45216,1433,T,I,D"]}]},{"rule":"UserRule_AllowInternetOut","flows":[{"mac":"000D3A97A1BA","flowTuples":["1547075524,172.20.0.132,52.239.177.196,14674,443,T,O,A","1547075524,172.20.0.132,52.239.177.196,14675,443,T,O,A","1547075524,172.20.0.132,52.239.177.196,14676,443,T,O,A","1547075550,172.20.0.132,52.239.177.196,14678,443,T,O,A","1547075550,172.20.0.132,52.239.177.196,14679,443,T,O,A","1547075550,172.20.0.132,52.239.177.196,14680,443,T,O,A"]}]}]}}
{"time":"2019-01-09T23:11:42.9066191Z","systemId":"4ed3b8cc-3bd0-4e6f-ae79-ffeea35a858d","category":"NetworkSecurityGroupFlowEvent","resourceId":"/SUBSCRIPTIONS/2E6A794A-489A-4824-AB75-DCDC4D0ECDC5/RESOURCEGROUPS/CENTRAL_RG/PROVIDERS/MICROSOFT.NETWORK/NETWORKSECURITYGROUPS/NSG-POC-PUBLIC","operationName":"NetworkSecurityGroupFlowEvents","properties":{"Version":1,"flows":[{"rule":"UserRule_DefaultInboundDenyAll","flows":[{"mac":"000D3A97A1BA","flowTuples":["1547075455,185.255.31.2,172.20.0.132,35081,1137,T,I,D","1547075461,111.35.172.147,172.20.0.132,39603,23,T,I,D","1547075464,149.248.3.183,172.20.0.132,56684,3342,T,I,D","1547075472,178.128.45.71,172.20.0.132,55955,8088,T,I,D","1547075473,185.244.25.108,172.20.0.132,60308,8088,T,I,D"]}]},{"rule":"UserRule_AllowInternetOut","flows":[{"mac":"000D3A97A1BA","flowTuples":["1547075448,172.20.0.132,52.239.177.196,14662,443,T,O,A","1547075448,172.20.0.132,52.239.177.196,14663,443,T,O,A","1547075448,172.20.0.132,52.239.177.196,14664,443,T,O,A","1547075473,172.20.0.132,52.239.177.196,14666,443,T,O,A","1547075473,172.20.0.132,52.239.177.196,14667,443,T,O,A","1547075473,172.20.0.132,52.239.177.196,14668,443,T,O,A","1547075499,172.20.0.132,52.239.177.196,14670,443,T,O,A","1547075499,172.20.0.132,52.239.177.196,14671,443,T,O,A","1547075499,172.20.0.132,52.239.177.196,14672,443,T,O,A"]}]}]}}
... View more