How to parse NSG Flow Log?

[Rhidian]

Hi,

I'm trying to onboard NSG Flow Logs and while I have managed to break the events into the specific tuples as per this link [https://answers.splunk.com/answers/714696/process-json-azure-nsg-flow-log-tuples.html?_ga=2.12328442... I lose a lot of useful information that I need such as "rule" does anyone have any ideas?

{ "records": [ { "time": "2017-02-16T22:00:32.8950000Z", "systemId": "2c002c16-72f3-4dc5-b391-3444c3527434", "category": "NetworkSecurityGroupFlowEvent", "resourceId": "/SUBSCRIPTIONS/00000000-0000-0000-0000-000000000000/RESOURCEGROUPS/FABRIKAMRG/PROVIDERS/MICROSOFT.NETWORK/NETWORKSECURITYGROUPS/FABRIAKMVM1-NSG", "operationName": "NetworkSecurityGroupFlowEvents", "properties": { "Version": 1, "flows": [ { "rule": "DefaultRule_DenyAllInBound", "flows": [ { "mac": "000D3AF8801A", "flowTuples": [ "1487282421,42.119.146.95,10.1.0.4,51529,5358,T,I,D" ] } ] }, { "rule": "UserRule_default-allow-rdp", "flows": [ { "mac": "000D3AF8801A", "flowTuples": [ "1487282370,163.28.66.17,10.1.0.4,61771,3389,T,I,A", "1487282393,5.39.218.34,10.1.0.4,58596,3389,T,I,A", "1487282393,91.224.160.154,10.1.0.4,61540,3389,T,I,A", "1487282423,13.76.89.229,10.1.0.4,53163,3389,T,I,A" ] } ] } ] } }, { "time": "2017-02-16T22:01:32.8960000Z", "systemId": "2c002c16-72f3-4dc5-b391-3444c3527434", "category": "NetworkSecurityGroupFlowEvent", "resourceId": "/SUBSCRIPTIONS/00000000-0000-0000-0000-000000000000/RESOURCEGROUPS/FABRIKAMRG/PROVIDERS/MICROSOFT.NETWORK/NETWORKSECURITYGROUPS/FABRIAKMVM1-NSG", "operationName": "NetworkSecurityGroupFlowEvents", "properties": { "Version": 1, "flows": [ { "rule": "DefaultRule_DenyAllInBound", "flows": [ { "mac": "000D3AF8801A", "flowTuples": [ "1487282481,195.78.210.194,10.1.0.4,53,1732,U,I,D" ] } ] }, { "rule": "UserRule_default-allow-rdp", "flows": [ { "mac": "000D3AF8801A", "flowTuples": [ "1487282435,61.129.251.68,10.1.0.4,57776,3389,T,I,A", "1487282454,84.25.174.170,10.1.0.4,59085,3389,T,I,A", "1487282477,77.68.9.50,10.1.0.4,65078,3389,T,I,A" ] } ] } ] } }, "records": [ { "time": "2017-02-16T22:00:32.8950000Z", "systemId": "2c002c16-72f3-4dc5-b391-3444c3527434", "category": "NetworkSecurityGroupFlowEvent", "resourceId": "/SUBSCRIPTIONS/00000000-0000-0000-0000-000000000000/RESOURCEGROUPS/FABRIKAMRG/PROVIDERS/MICROSOFT.NETWORK/NETWORKSECURITYGROUPS/FABRIAKMVM1-NSG", "operationName": "NetworkSecurityGroupFlowEvents", "properties": {"Version":1,"flows":[{"rule":"DefaultRule_DenyAllInBound","flows":[{"mac":"000D3AF8801A","flowTuples":["1487282421,42.119.146.95,10.1.0.4,51529,5358,T,I,D"]}]},{"rule":"UserRule_default-allow-rdp","flows":[{"mac":"000D3AF8801A","flowTuples":["1487282370,163.28.66.17,10.1.0.4,61771,3389,T,I,A","1487282393,5.39.218.34,10.1.0.4,58596,3389,T,I,A","1487282393,91.224.160.154,10.1.0.4,61540,3389,T,I,A","1487282423,13.76.89.229,10.1.0.4,53163,3389,T,I,A"]}]}]} } , { "time": "2017-02-16T22:01:32.8960000Z", "systemId": "2c002c16-72f3-4dc5-b391-3444c3527434", "category": "NetworkSecurityGroupFlowEvent", "resourceId": "/SUBSCRIPTIONS/00000000-0000-0000-0000-000000000000/RESOURCEGROUPS/FABRIKAMRG/PROVIDERS/MICROSOFT.NETWORK/NETWORKSECURITYGROUPS/FABRIAKMVM1-NSG", "operationName": "NetworkSecurityGroupFlowEvents", "properties": {"Version":1,"flows":[{"rule":"DefaultRule_DenyAllInBound","flows":[{"mac":"000D3AF8801A","flowTuples":["1487282481,195.78.210.194,10.1.0.4,53,1732,U,I,D"]}]},{"rule":"UserRule_default-allow-rdp","flows":[{"mac":"000D3AF8801A","flowTuples":["1487282435,61.129.251.68,10.1.0.4,57776,3389,T,I,A","1487282454,84.25.174.170,10.1.0.4,59085,3389,T,I,A","1487282477,77.68.9.50,10.1.0.4,65078,3389,T,I,A"]}]}]} } , { "time": "2017-02-16T22:02:32.9040000Z", "systemId": "2c002c16-72f3-4dc5-b391-3444c3527434", "category": "NetworkSecurityGroupFlowEvent", "resourceId": "/SUBSCRIPTIONS/00000000-0000-0000-0000-000000000000/RESOURCEGROUPS/FABRIKAMRG/PROVIDERS/MICROSOFT.NETWORK/NETWORKSECURITYGROUPS/FABRIAKMVM1-NSG", "operationName": "NetworkSecurityGroupFlowEvents", "properties": {"Version":1,"flows":[{"rule":"DefaultRule_DenyAllInBound","flows":[{"mac":"000D3AF8801A","flowTuples":["1487282492,175.182.69.29,10.1.0.4,28918,5358,T,I,D","1487282505,71.6.216.55,10.1.0.4,8080,8080,T,I,D"]}]},{"rule":"UserRule_default-allow-rdp","flows":[{"mac":"000D3AF8801A","flowTuples":["1487282512,91.224.160.154,10.1.0.4,59046,3389,T,I,A"]}]}]} }

[richgalloway]

Have you tried the spath command?

[Rhidian]

I thought of spat but I'm limited in what I can do as we use Splun Cloud. If I understand it I can use KV_MODE = xml and hopefully the parsing will be OK and do the rest through spath in the search bar. I'm not 100% convinced this is compliant XML especially when it get to the tuples as there are these are mulitline and don't have individual key pairs. I will try it later and see what happens.

[richgalloway]

I copied your example event into a search and spath seemed to parse it OK. 

You can use KV_MODE=xml, but that's discouraged because it causes a lot of fields to be extracted at index-time, taking up resources.  Better to use spath at search time.  It may be necessary to extract the legit JSON from the raw event before calling spath.