Getting Data In

How to parse NSG Flow Log?

Rhidian
Path Finder

Hi,

I'm trying to onboard NSG Flow Logs and while I have managed to break the events into the specific tuples as per this link [https://answers.splunk.com/answers/714696/process-json-azure-nsg-flow-log-tuples.html?_ga=2.12328442... I lose a lot of useful information that I need such as "rule" does anyone have any ideas?

{ "records": [ { "time": "2017-02-16T22:00:32.8950000Z", "systemId": "2c002c16-72f3-4dc5-b391-3444c3527434", "category": "NetworkSecurityGroupFlowEvent", "resourceId": "/SUBSCRIPTIONS/00000000-0000-0000-0000-000000000000/RESOURCEGROUPS/FABRIKAMRG/PROVIDERS/MICROSOFT.NETWORK/NETWORKSECURITYGROUPS/FABRIAKMVM1-NSG", "operationName": "NetworkSecurityGroupFlowEvents", "properties": { "Version": 1, "flows": [ { "rule": "DefaultRule_DenyAllInBound", "flows": [ { "mac": "000D3AF8801A", "flowTuples": [ "1487282421,42.119.146.95,10.1.0.4,51529,5358,T,I,D" ] } ] }, { "rule": "UserRule_default-allow-rdp", "flows": [ { "mac": "000D3AF8801A", "flowTuples": [ "1487282370,163.28.66.17,10.1.0.4,61771,3389,T,I,A", "1487282393,5.39.218.34,10.1.0.4,58596,3389,T,I,A", "1487282393,91.224.160.154,10.1.0.4,61540,3389,T,I,A", "1487282423,13.76.89.229,10.1.0.4,53163,3389,T,I,A" ] } ] } ] } }, { "time": "2017-02-16T22:01:32.8960000Z", "systemId": "2c002c16-72f3-4dc5-b391-3444c3527434", "category": "NetworkSecurityGroupFlowEvent", "resourceId": "/SUBSCRIPTIONS/00000000-0000-0000-0000-000000000000/RESOURCEGROUPS/FABRIKAMRG/PROVIDERS/MICROSOFT.NETWORK/NETWORKSECURITYGROUPS/FABRIAKMVM1-NSG", "operationName": "NetworkSecurityGroupFlowEvents", "properties": { "Version": 1, "flows": [ { "rule": "DefaultRule_DenyAllInBound", "flows": [ { "mac": "000D3AF8801A", "flowTuples": [ "1487282481,195.78.210.194,10.1.0.4,53,1732,U,I,D" ] } ] }, { "rule": "UserRule_default-allow-rdp", "flows": [ { "mac": "000D3AF8801A", "flowTuples": [ "1487282435,61.129.251.68,10.1.0.4,57776,3389,T,I,A", "1487282454,84.25.174.170,10.1.0.4,59085,3389,T,I,A", "1487282477,77.68.9.50,10.1.0.4,65078,3389,T,I,A" ] } ] } ] } }, "records": [ { "time": "2017-02-16T22:00:32.8950000Z", "systemId": "2c002c16-72f3-4dc5-b391-3444c3527434", "category": "NetworkSecurityGroupFlowEvent", "resourceId": "/SUBSCRIPTIONS/00000000-0000-0000-0000-000000000000/RESOURCEGROUPS/FABRIKAMRG/PROVIDERS/MICROSOFT.NETWORK/NETWORKSECURITYGROUPS/FABRIAKMVM1-NSG", "operationName": "NetworkSecurityGroupFlowEvents", "properties": {"Version":1,"flows":[{"rule":"DefaultRule_DenyAllInBound","flows":[{"mac":"000D3AF8801A","flowTuples":["1487282421,42.119.146.95,10.1.0.4,51529,5358,T,I,D"]}]},{"rule":"UserRule_default-allow-rdp","flows":[{"mac":"000D3AF8801A","flowTuples":["1487282370,163.28.66.17,10.1.0.4,61771,3389,T,I,A","1487282393,5.39.218.34,10.1.0.4,58596,3389,T,I,A","1487282393,91.224.160.154,10.1.0.4,61540,3389,T,I,A","1487282423,13.76.89.229,10.1.0.4,53163,3389,T,I,A"]}]}]} } , { "time": "2017-02-16T22:01:32.8960000Z", "systemId": "2c002c16-72f3-4dc5-b391-3444c3527434", "category": "NetworkSecurityGroupFlowEvent", "resourceId": "/SUBSCRIPTIONS/00000000-0000-0000-0000-000000000000/RESOURCEGROUPS/FABRIKAMRG/PROVIDERS/MICROSOFT.NETWORK/NETWORKSECURITYGROUPS/FABRIAKMVM1-NSG", "operationName": "NetworkSecurityGroupFlowEvents", "properties": {"Version":1,"flows":[{"rule":"DefaultRule_DenyAllInBound","flows":[{"mac":"000D3AF8801A","flowTuples":["1487282481,195.78.210.194,10.1.0.4,53,1732,U,I,D"]}]},{"rule":"UserRule_default-allow-rdp","flows":[{"mac":"000D3AF8801A","flowTuples":["1487282435,61.129.251.68,10.1.0.4,57776,3389,T,I,A","1487282454,84.25.174.170,10.1.0.4,59085,3389,T,I,A","1487282477,77.68.9.50,10.1.0.4,65078,3389,T,I,A"]}]}]} } , { "time": "2017-02-16T22:02:32.9040000Z", "systemId": "2c002c16-72f3-4dc5-b391-3444c3527434", "category": "NetworkSecurityGroupFlowEvent", "resourceId": "/SUBSCRIPTIONS/00000000-0000-0000-0000-000000000000/RESOURCEGROUPS/FABRIKAMRG/PROVIDERS/MICROSOFT.NETWORK/NETWORKSECURITYGROUPS/FABRIAKMVM1-NSG", "operationName": "NetworkSecurityGroupFlowEvents", "properties": {"Version":1,"flows":[{"rule":"DefaultRule_DenyAllInBound","flows":[{"mac":"000D3AF8801A","flowTuples":["1487282492,175.182.69.29,10.1.0.4,28918,5358,T,I,D","1487282505,71.6.216.55,10.1.0.4,8080,8080,T,I,D"]}]},{"rule":"UserRule_default-allow-rdp","flows":[{"mac":"000D3AF8801A","flowTuples":["1487282512,91.224.160.154,10.1.0.4,59046,3389,T,I,A"]}]}]} }

Labels (2)
0 Karma

richgalloway
SplunkTrust
SplunkTrust

Have you tried the spath command?

---
If this reply helps you, Karma would be appreciated.
0 Karma

Rhidian
Path Finder

I thought of spat but I'm limited in what I can do as we use Splun Cloud. If I understand it I can use KV_MODE = xml and hopefully the parsing will be OK and do the rest through spath in the search bar. I'm not 100% convinced this is compliant XML especially when it get to the tuples as there are these are mulitline and don't have individual key pairs. I will try it later and see what happens.

0 Karma

richgalloway
SplunkTrust
SplunkTrust

I copied your example event into a search and spath seemed to parse it OK. 

You can use KV_MODE=xml, but that's discouraged because it causes a lot of fields to be extracted at index-time, taking up resources.  Better to use spath at search time.  It may be necessary to extract the legit JSON from the raw event before calling spath.

---
If this reply helps you, Karma would be appreciated.
0 Karma
Get Updates on the Splunk Community!

Splunk Answers Content Calendar, June Edition

Get ready for this week’s post dedicated to Splunk Dashboards! We're celebrating the power of community by ...

What You Read The Most: Splunk Lantern’s Most Popular Articles!

Splunk Lantern is a Splunk customer success center that provides advice from Splunk experts on valuable data ...

See your relevant APM services, dashboards, and alerts in one place with the updated ...

As a Splunk Observability user, you have a lot of data you have to manage, prioritize, and troubleshoot on a ...