All Apps and Add-ons

Process JSON Azure NSG Flow Log Tuples

Communicator

I used @jconger's article Splunking Microsoft Azure Network Watcher Data to configure the Splunk Add-on for Microsoft Cloud Services to ingest Azure Network Security Group (NSG) flow logs. @jconger also provided a useful props.conf to clean up the JSON blobs that come in.

However, the JSON objects contain one or more tuples (connection flows), which result in having multiple srcips, destip, etc. in a single event. I need to separate each tuple into its own event so I can map the events into the Network Traffic data model. One thought is to use SEDCMD to remove everything but the tuples; this has two caveats: (1) I'm not sure how to create an effective RegEx for this, and (2) I lose fields such as "rule" and "resourceID" (contains NSG name) that apply to all tuples in a single JSON object. Here are some sample logs:

{"time":"2019-01-09T23:14:42.9159948Z","systemId":"4ed3b8cc-3bd0-4e6f-ae79-ffeea35a858d","category":"NetworkSecurityGroupFlowEvent","resourceId":"/SUBSCRIPTIONS/2E6A794A-489A-4824-AB75-DCDC4D0ECDC5/RESOURCEGROUPS/CENTRAL_RG/PROVIDERS/MICROSOFT.NETWORK/NETWORKSECURITYGROUPS/NSG-POC-PUBLIC","operationName":"NetworkSecurityGroupFlowEvents","properties":{"Version":1,"flows":[{"rule":"UserRule_DefaultInboundDenyAll","flows":[{"mac":"000D3A97A1BA","flowTuples":["1547075623,104.248.168.1,172.20.0.132,51353,8088,T,I,D"]}]},{"rule":"UserRule_AllowInternetOut","flows":[{"mac":"000D3A97A1BA","flowTuples":["1547075626,172.20.0.132,52.239.177.196,14690,443,T,O,A","1547075626,172.20.0.132,52.239.177.196,14691,443,T,O,A","1547075626,172.20.0.132,52.239.177.196,14692,443,T,O,A","1547075628,172.20.0.132,208.91.113.71,123,123,U,O,A","1547075628,172.20.0.132,208.91.112.51,123,123,U,O,A","1547075628,172.20.0.132,208.91.112.50,123,123,U,O,A","1547075634,172.20.0.132,208.91.113.70,123,123,U,O,A","1547075652,172.20.0.132,52.239.177.196,14694,443,T,O,A","1547075652,172.20.0.132,52.239.177.196,14695,443,T,O,A","1547075652,172.20.0.132,52.239.177.196,14696,443,T,O,A","1547075677,172.20.0.132,52.239.177.196,14698,443,T,O,A","1547075677,172.20.0.132,52.239.177.196,14699,443,T,O,A","1547075677,172.20.0.132,52.239.177.196,14700,443,T,O,A"]}]}]}}
{"time":"2019-01-09T23:13:42.9136121Z","systemId":"4ed3b8cc-3bd0-4e6f-ae79-ffeea35a858d","category":"NetworkSecurityGroupFlowEvent","resourceId":"/SUBSCRIPTIONS/2E6A794A-489A-4824-AB75-DCDC4D0ECDC5/RESOURCEGROUPS/CENTRAL_RG/PROVIDERS/MICROSOFT.NETWORK/NETWORKSECURITYGROUPS/NSG-POC-PUBLIC","operationName":"NetworkSecurityGroupFlowEvents","properties":{"Version":1,"flows":[{"rule":"UserRule_DefaultInboundDenyAll","flows":[{"mac":"000D3A97A1BA","flowTuples":["1547075589,194.28.115.245,172.20.0.132,47661,33897,T,I,D","1547075614,178.128.44.249,172.20.0.132,59511,8088,T,I,D"]}]},{"rule":"UserRule_AllowInternetOut","flows":[{"mac":"000D3A97A1BA","flowTuples":["1547075575,172.20.0.132,52.239.177.196,14682,443,T,O,A","1547075575,172.20.0.132,52.239.177.196,14683,443,T,O,A","1547075575,172.20.0.132,52.239.177.196,14684,443,T,O,A","1547075601,172.20.0.132,52.239.177.196,14686,443,T,O,A","1547075601,172.20.0.132,52.239.177.196,14687,443,T,O,A","1547075601,172.20.0.132,52.239.177.196,14688,443,T,O,A"]}]}]}}
{"time":"2019-01-09T23:12:42.9104712Z","systemId":"4ed3b8cc-3bd0-4e6f-ae79-ffeea35a858d","category":"NetworkSecurityGroupFlowEvent","resourceId":"/SUBSCRIPTIONS/2E6A794A-489A-4824-AB75-DCDC4D0ECDC5/RESOURCEGROUPS/CENTRAL_RG/PROVIDERS/MICROSOFT.NETWORK/NETWORKSECURITYGROUPS/NSG-POC-PUBLIC","operationName":"NetworkSecurityGroupFlowEvents","properties":{"Version":1,"flows":[{"rule":"UserRule_DefaultInboundDenyAll","flows":[{"mac":"000D3A97A1BA","flowTuples":["1547075521,165.76.160.44,172.20.0.132,45216,1433,T,I,D"]}]},{"rule":"UserRule_AllowInternetOut","flows":[{"mac":"000D3A97A1BA","flowTuples":["1547075524,172.20.0.132,52.239.177.196,14674,443,T,O,A","1547075524,172.20.0.132,52.239.177.196,14675,443,T,O,A","1547075524,172.20.0.132,52.239.177.196,14676,443,T,O,A","1547075550,172.20.0.132,52.239.177.196,14678,443,T,O,A","1547075550,172.20.0.132,52.239.177.196,14679,443,T,O,A","1547075550,172.20.0.132,52.239.177.196,14680,443,T,O,A"]}]}]}}
{"time":"2019-01-09T23:11:42.9066191Z","systemId":"4ed3b8cc-3bd0-4e6f-ae79-ffeea35a858d","category":"NetworkSecurityGroupFlowEvent","resourceId":"/SUBSCRIPTIONS/2E6A794A-489A-4824-AB75-DCDC4D0ECDC5/RESOURCEGROUPS/CENTRAL_RG/PROVIDERS/MICROSOFT.NETWORK/NETWORKSECURITYGROUPS/NSG-POC-PUBLIC","operationName":"NetworkSecurityGroupFlowEvents","properties":{"Version":1,"flows":[{"rule":"UserRule_DefaultInboundDenyAll","flows":[{"mac":"000D3A97A1BA","flowTuples":["1547075455,185.255.31.2,172.20.0.132,35081,1137,T,I,D","1547075461,111.35.172.147,172.20.0.132,39603,23,T,I,D","1547075464,149.248.3.183,172.20.0.132,56684,3342,T,I,D","1547075472,178.128.45.71,172.20.0.132,55955,8088,T,I,D","1547075473,185.244.25.108,172.20.0.132,60308,8088,T,I,D"]}]},{"rule":"UserRule_AllowInternetOut","flows":[{"mac":"000D3A97A1BA","flowTuples":["1547075448,172.20.0.132,52.239.177.196,14662,443,T,O,A","1547075448,172.20.0.132,52.239.177.196,14663,443,T,O,A","1547075448,172.20.0.132,52.239.177.196,14664,443,T,O,A","1547075473,172.20.0.132,52.239.177.196,14666,443,T,O,A","1547075473,172.20.0.132,52.239.177.196,14667,443,T,O,A","1547075473,172.20.0.132,52.239.177.196,14668,443,T,O,A","1547075499,172.20.0.132,52.239.177.196,14670,443,T,O,A","1547075499,172.20.0.132,52.239.177.196,14671,443,T,O,A","1547075499,172.20.0.132,52.239.177.196,14672,443,T,O,A"]}]}]}}

Communicator

So I may have stumbled across a solution. It starts with setting the LINE_BREAKER right before the epoch time of each tuple:

LINE_BREAKER = (\")\d{10}

This will start every event with the tuple. Then, I use SEDCMD to remove strings that begin with quotes through the end of the line, as long as the character after the quote wasn't a digit (to exclude tuples, because they begin with a digit, because it's epoch time):

SEDCMD-removenotepoch = s/\"\D.*$//g

Now all my logs are CSV tuples! I lose some information from the JSON object like rule name, but I can grab the NSG name from "source" because it's in the source directory. This is the best solution I've come across for our environment - please let me know if anyone sees any caveats.

0 Karma

New Member

Would you be able to provide syntax in transform and props files?

I am having a similar issue and I not sure where the LINE_BREAKER or SEDCMD would go in file.

0 Karma

Communicator

Sure. I named my sourcetype "mscs:nsg:flow" when I configured the Splunk Add-on for Microsoft Cloud Services.

These configurations will need to be on the Splunk instance ingesting the data.

props.conf
[mscs:nsg:flow]
LINE_BREAKER = (\")\d{10}
SHOULD_LINEMERGE = false
SEDCMD-remove_not_epoch = s/\"\D.*$//g
0 Karma

New Member

Thank you, I think found another post of yours. I've replicated, in my environment with success.

Many thanks!!

You comment here from Jan 17th
https://answers.splunk.com/answers/666758/import-azure-nsg-logs.html

0 Karma