How to process JSON Azure NSG Flow Log Tuples?

[ejwade]

I used @jconger's article Splunking Microsoft Azure Network Watcher Data to configure the Splunk Add-on for Microsoft Cloud Services to ingest Azure Network Security Group (NSG) flow logs. @jconger also provided a useful props.conf to clean up the JSON blobs that come in.

However, the JSON objects contain one or more tuples (connection flows), which result in having multiple src_ips, dest_ip, etc. in a single event. I need to separate each tuple into its own event so I can map the events into the Network Traffic data model. One thought is to use SEDCMD to remove everything but the tuples; this has two caveats: (1) I'm not sure how to create an effective RegEx for this, and (2) I lose fields such as "rule" and "resourceID" (contains NSG name) that apply to all tuples in a single JSON object. Here are some sample logs:

{"time":"2019-01-09T23:14:42.9159948Z","systemId":"4ed3b8cc-3bd0-4e6f-ae79-ffeea35a858d","category":"NetworkSecurityGroupFlowEvent","resourceId":"/SUBSCRIPTIONS/2E6A794A-489A-4824-AB75-DCDC4D0ECDC5/RESOURCEGROUPS/CENTRAL_RG/PROVIDERS/MICROSOFT.NETWORK/NETWORKSECURITYGROUPS/NSG-POC-PUBLIC","operationName":"NetworkSecurityGroupFlowEvents","properties":{"Version":1,"flows":[{"rule":"UserRule_DefaultInboundDenyAll","flows":[{"mac":"000D3A97A1BA","flowTuples":["1547075623,104.248.168.1,172.20.0.132,51353,8088,T,I,D"]}]},{"rule":"UserRule_AllowInternetOut","flows":[{"mac":"000D3A97A1BA","flowTuples":["1547075626,172.20.0.132,52.239.177.196,14690,443,T,O,A","1547075626,172.20.0.132,52.239.177.196,14691,443,T,O,A","1547075626,172.20.0.132,52.239.177.196,14692,443,T,O,A","1547075628,172.20.0.132,208.91.113.71,123,123,U,O,A","1547075628,172.20.0.132,208.91.112.51,123,123,U,O,A","1547075628,172.20.0.132,208.91.112.50,123,123,U,O,A","1547075634,172.20.0.132,208.91.113.70,123,123,U,O,A","1547075652,172.20.0.132,52.239.177.196,14694,443,T,O,A","1547075652,172.20.0.132,52.239.177.196,14695,443,T,O,A","1547075652,172.20.0.132,52.239.177.196,14696,443,T,O,A","1547075677,172.20.0.132,52.239.177.196,14698,443,T,O,A","1547075677,172.20.0.132,52.239.177.196,14699,443,T,O,A","1547075677,172.20.0.132,52.239.177.196,14700,443,T,O,A"]}]}]}}
{"time":"2019-01-09T23:13:42.9136121Z","systemId":"4ed3b8cc-3bd0-4e6f-ae79-ffeea35a858d","category":"NetworkSecurityGroupFlowEvent","resourceId":"/SUBSCRIPTIONS/2E6A794A-489A-4824-AB75-DCDC4D0ECDC5/RESOURCEGROUPS/CENTRAL_RG/PROVIDERS/MICROSOFT.NETWORK/NETWORKSECURITYGROUPS/NSG-POC-PUBLIC","operationName":"NetworkSecurityGroupFlowEvents","properties":{"Version":1,"flows":[{"rule":"UserRule_DefaultInboundDenyAll","flows":[{"mac":"000D3A97A1BA","flowTuples":["1547075589,194.28.115.245,172.20.0.132,47661,33897,T,I,D","1547075614,178.128.44.249,172.20.0.132,59511,8088,T,I,D"]}]},{"rule":"UserRule_AllowInternetOut","flows":[{"mac":"000D3A97A1BA","flowTuples":["1547075575,172.20.0.132,52.239.177.196,14682,443,T,O,A","1547075575,172.20.0.132,52.239.177.196,14683,443,T,O,A","1547075575,172.20.0.132,52.239.177.196,14684,443,T,O,A","1547075601,172.20.0.132,52.239.177.196,14686,443,T,O,A","1547075601,172.20.0.132,52.239.177.196,14687,443,T,O,A","1547075601,172.20.0.132,52.239.177.196,14688,443,T,O,A"]}]}]}}
{"time":"2019-01-09T23:12:42.9104712Z","systemId":"4ed3b8cc-3bd0-4e6f-ae79-ffeea35a858d","category":"NetworkSecurityGroupFlowEvent","resourceId":"/SUBSCRIPTIONS/2E6A794A-489A-4824-AB75-DCDC4D0ECDC5/RESOURCEGROUPS/CENTRAL_RG/PROVIDERS/MICROSOFT.NETWORK/NETWORKSECURITYGROUPS/NSG-POC-PUBLIC","operationName":"NetworkSecurityGroupFlowEvents","properties":{"Version":1,"flows":[{"rule":"UserRule_DefaultInboundDenyAll","flows":[{"mac":"000D3A97A1BA","flowTuples":["1547075521,165.76.160.44,172.20.0.132,45216,1433,T,I,D"]}]},{"rule":"UserRule_AllowInternetOut","flows":[{"mac":"000D3A97A1BA","flowTuples":["1547075524,172.20.0.132,52.239.177.196,14674,443,T,O,A","1547075524,172.20.0.132,52.239.177.196,14675,443,T,O,A","1547075524,172.20.0.132,52.239.177.196,14676,443,T,O,A","1547075550,172.20.0.132,52.239.177.196,14678,443,T,O,A","1547075550,172.20.0.132,52.239.177.196,14679,443,T,O,A","1547075550,172.20.0.132,52.239.177.196,14680,443,T,O,A"]}]}]}}
{"time":"2019-01-09T23:11:42.9066191Z","systemId":"4ed3b8cc-3bd0-4e6f-ae79-ffeea35a858d","category":"NetworkSecurityGroupFlowEvent","resourceId":"/SUBSCRIPTIONS/2E6A794A-489A-4824-AB75-DCDC4D0ECDC5/RESOURCEGROUPS/CENTRAL_RG/PROVIDERS/MICROSOFT.NETWORK/NETWORKSECURITYGROUPS/NSG-POC-PUBLIC","operationName":"NetworkSecurityGroupFlowEvents","properties":{"Version":1,"flows":[{"rule":"UserRule_DefaultInboundDenyAll","flows":[{"mac":"000D3A97A1BA","flowTuples":["1547075455,185.255.31.2,172.20.0.132,35081,1137,T,I,D","1547075461,111.35.172.147,172.20.0.132,39603,23,T,I,D","1547075464,149.248.3.183,172.20.0.132,56684,3342,T,I,D","1547075472,178.128.45.71,172.20.0.132,55955,8088,T,I,D","1547075473,185.244.25.108,172.20.0.132,60308,8088,T,I,D"]}]},{"rule":"UserRule_AllowInternetOut","flows":[{"mac":"000D3A97A1BA","flowTuples":["1547075448,172.20.0.132,52.239.177.196,14662,443,T,O,A","1547075448,172.20.0.132,52.239.177.196,14663,443,T,O,A","1547075448,172.20.0.132,52.239.177.196,14664,443,T,O,A","1547075473,172.20.0.132,52.239.177.196,14666,443,T,O,A","1547075473,172.20.0.132,52.239.177.196,14667,443,T,O,A","1547075473,172.20.0.132,52.239.177.196,14668,443,T,O,A","1547075499,172.20.0.132,52.239.177.196,14670,443,T,O,A","1547075499,172.20.0.132,52.239.177.196,14671,443,T,O,A","1547075499,172.20.0.132,52.239.177.196,14672,443,T,O,A"]}]}]}}

[ejwade]

So I may have stumbled across a solution. It starts with setting the LINE_BREAKER right before the epoch time of each tuple:

LINE_BREAKER = (\")\d{10}

This will start every event with the tuple. Then, I use SEDCMD to remove strings that begin with quotes through the end of the line, as long as the character after the quote wasn't a digit (to exclude tuples, because they begin with a digit, because it's epoch time):

SEDCMD-remove_not_epoch = s/\"\D.*$//g

Now all my logs are CSV tuples! I lose some information from the JSON object like rule name, but I can grab the NSG name from "source" because it's in the source directory. This is the best solution I've come across for our environment - please let me know if anyone sees any caveats.

[SteveO86]

Would you be able to provide syntax in transform and props files?

I am having a similar issue and I not sure where the LINE_BREAKER or SEDCMD would go in file.

[ejwade]

Sure. I named my sourcetype "mscs:nsg:flow" when I configured the Splunk Add-on for Microsoft Cloud Services.

These configurations will need to be on the Splunk instance ingesting the data.

props.conf
[mscs:nsg:flow]
LINE_BREAKER = (\")\d{10}
SHOULD_LINEMERGE = false
SEDCMD-remove_not_epoch = s/\"\D.*$//g

[SteveO86]

Thank you, I think found another post of yours. I've replicated, in my environment with success.

Many thanks!!

You comment here from Jan 17th
https://answers.splunk.com/answers/666758/import-azure-nsg-logs.html

[Rhidian]

Apologies for reviving this thread but I have implemented what has been suggested and my data still looks terrible. Can someone please post an example of what it should look like?