NSG Logs into Splunk - Props & Transforms Issue

[anandhalagarasa]

Hi Team,

We got an requirement to ingest NSG logs from Azure into Splunk Cloud. So we have installed the "Splunk Add-on for Microsoft Cloud Services" in our Heavy Forwarder server and we have created the inputs in Azure Storage Blob and post which I can able to see the logs were getting ingested into Splunk Cloud but a single event seems to be a combination of multiple events.

So I have tried this solution by applying the props and transforms.conf as mentioned below:

https://www.splunk.com/blog/2017/02/20/splunking-microsoft-azure-network-watcher-data.html

props.conf:

[sourcetype]
LINE_BREAKER = }([\r\n]s*,[\r\n]s*){
SEDCMD-remove_header = s/{\s*\"records\"\:s*[\s*//g
SEDCMD-remove_footer = s/][\r\n]\s*}.*//g
SHOULD_LINEMERGE = false
KV_MODE = json
TIME_PREFIX = time\":\"
REPORT-tuples = extract_tuple

transforms.conf:

[extract_tuple]
SOURCE_KEY = properties.flows{}.flows{}.flowTuples{}
DELIMS = ","
FIELDS = time,src_ip,dst_ip,src_port,dst_port,protocol,traffic_flow,traffic_result

Then I have tried the Splunk answers Regex provided in the following weburl:

https://answers.splunk.com/answers/714696/process-json-azure-nsg-flow-log-tuples.html

props.conf:

[sourcetype]
LINE_BREAKER = (\")\d{10}
SHOULD_LINEMERGE = false
SEDCMD-remove_not_epoch = s/\"\D.*$//g

transforms.conf:

[extract_tuple]
DELIMS = ","
FIELDS = time,src_ip,dest_ip,src_port,dest_port,transport,traffic_flow,traffic_result

But still the logs are getting generated as a very big event and after 10000 characters its getting categorized as a single event so kindly help to provide relevant props and transforms.conf so that I can implement in my environment and check the feasibility of it.

[anandhalagarasa]

can anyone help on the request.

[tbavarva]

Hi Anand,

Can you please provide any sample log to test regex?

Regards,
Tejas

[anandhalagarasa]

As requested a single event looks like as below.

x,xx.xxx.xx.x,xx.xxx.x.x,xxxxx,xxx,T,O,A,B,,,,","xxxxxxxxxx,xx.xxx.xx.x,xx.xxx.x.x,xxxxx,xxx,T,O,A,E,,,,","xxxxxxxxxx,xx.xxx.xx.x,xx.xxx.x.x,xxxxx,xxx,T,O,A,B,,,,","xxxxxxxxxx,xx.xxx.xx.x,xx.xxx.x.x,xxxxx,xxx,T,O,A,E,,,,","xxxxxxxxxx,xx.xxx.xx.x,xx.xxx.x.x,xxxxx,xxx,T,O,A,B,,,,","xxxxxxxxxx,xx.xxx.xx.x,xx.xxx.x.xx,xxxxx,xxxx,T,O,A,B,,,,","xxxxxxxxxx,xx.xxx.xx.x,xx.xxx.xxx.xxx,xxxxx,xxxx,T,O,A,E,,,,","xxxxxxxxxx,xx.xxx.xx.x,xx.xxx.x.x,xxxxx,xxx,T,O,A,E,,,,","xxxxxxxxxx,xx.xxx.xx.x,xx.xxx.x.x,xxxxx,xxx,T,O,A,B,,,,","xxxxxxxxxx,xx.xxx.xx.x,xx.xxx.x.x,xxxxx,xxx,T,O,A,E,,,,","xxxxxxxxxx,xx.xxx.xx.x,xx.xxx.x.x,xxxxx,xxx,T,O,A,B,,,,","xxxxxxxxxx,xx.xxx.xx.x,x.xx.xx.xxx,xxxxx,xxxx,T,O,A,B,,,,","xxxxxxxxxx,xx.xxx.xx.x,xx.xxx.x.x,xxxxx,xxx,T,O,A,E,,,,","xxxxxxxxxx,xx.xxx.xx.x,xx.xxx.x.x,xxxxx,xxx,T,O,A,B,,,,","xxxxxxxxxx,xx.xxx.xx.x,xx.xxx.x.x,xxxxx,xxx,T,O,A,E,,,,","xxxxxxxxxx,xx.xxx.xx.x,x.xx.xx.xxx,xxxxx,xxxx,T,O,A,E,,,,","xxxxxxxxxx,xx.xxx.xx.x,xx.xxx.x.xx,xxxxx,xxxx,T,O,A,E,,,,"]}]}]}},{"time":"xxxx-xx-xxTxx:xx:xx.xxxxxxxZ","systemId":"bxxxexxx-xfxx-xbxe-becx-xxdfxxxxxxxx","macAddress":"xxxDxAxEExCx","category":"NetworkSecurityGroupFlowEvent","resourceId":"/SUBSCRIPTIONS/XXxXXXx-XXXx-xxxx-Xxxx-xXxxxxXXxXxX/RESOURCEGROUPS/MCSPUSxNETWORK/PROVIDERS/MICROSOFT.NETWORK/NETWORKSECURITYGROUPS/USxPORINDI","operationName":"NetworkSecurityGroupFlowEvents","properties":{"Version":x,"flows":[{"rule":"UserRule_servicechecks","flows":[{"mac":"xxxDxAxEExCx","flowTuples":["xxxxxxxxxx,xx.xxx.x.xxx,xx.xxx.xx.x,xxxxx,xxx,T,I,A,E,x,xxx,x,xxx","xxxxxxxxxx,xx.xxx.x.xxx,xx.xxx.xx.x,xxxxx,xxx,T,I,A,B,,,,","xxxxxxxxxx,xx.xxx.x.xxx,xx.xxx.xx.x,xxxxx,xxx,T,I,A,E,x,xxx,x,xxx","xxxxxxxxxx,xx.xxx.x.xxx,xx.xxx.xx.x,xxxxx,xxx,T,I,A,B,,,,","xxxxxxxxxx,xx.xxx.x.xxx,xx.xxx.xx.x,xxxxx,xxx,T,I,A,E,x,xxx,x,xxx","xxxxxxxxxx,xx.xxx.x.xxx,xx.xxx.xx.x,xxxxx,xxx,T,I,A,B,,,,","xxxxxxxxxx,xx.xxx.x.xxx,xx.xxx.xx.x,xxxxx,xxx,T,I,A,E,x,xxx,x,xxx","xxxxxxxxxx,xx.xxx.x.xxx,xx.xxx.xx.x,xxxxx,xxx,T,I,A,B,,,,","xxxxxxxxxx,xx.xxx.x.xxx,xx.xxx.xx.x,xxxxx,xxx,T,I,A,E,x,xxx,x,xxx","xxxxxxxxxx,xx.xxx.x.xxx,xx.xxx.xx.x,xxxxx,xxx,T,I,A,B,,,,","xxxxxxxxxx,xx.xxx.x.xxx,xx.xxx.xx.x,xxxxx,xxx,T,I,A,E,x,xxx,x,xxx","xxxxxxxxxx,xx.xxx.x.xxx,xx.xxx.xx.x,xxxxx,xxx,T,I,A,B,,,,","xxxxxxxxxx,xx.xxx.x.xxx,xx.xxx.xx.x,xxxxx,xxx,T,I,A,E,x,xxx,x,xxx","xxxxxxxxxx,xx.xxx.x.xxx,xx.xxx.xx.x,xxxxx,xxx,T,I,A,B,,,,","xxxxxxxxxx,xx.xxx.x.xxx,xx.xxx.xx.x,xxxxx,xxx,T,I,A,E,x,xxx,x,xxx","xxxxxxxxxx,xx.xxx.x.xxx,xx.xxx.xx.x,xxxxx,xxx,T,I,A,B,,,,"]}]},{"rule":"DefaultRule_AllowVnetOutBound","flows":[{"mac":"xxxDxAxEExCx","flowTuples":["xxxxxxxxxx,xx.xxx.xx.x,xx.xxx.xx.xx,xxxxx,xxxx,T,O,A,B,,,,","xxxxxxxxxx,xx.xxx.xx.x,xx.xxx.x.x,xxxxx,xxx,T,O,A,B,,,,","xxxxxxxxxx,xx.xxx.xx.x,xx.xxx.x.x,xxxxx,xxx,T,O,A,E,,,,","xxxxxxxxxx,xx.xxx.xx.x,xx.xxx.x.x,xxxxx,xxx,T,O,A,B,,,,","xxxxxxxxxx,xx.xxx.xx.x,xx.xxx.x.x,xxxxx,xxx,T,O,A,E,,,,","xxxxxxxxxx,xx.xxx.xx.x,xx.xxx.x.x,xxxxx,xxxxx,T,O,A,E,,,,","xxxxxxxxxx,xx.xxx.xx.x,xx.xxx.x.x,xxxxx,xxx,T,O,A,E,,,,","xxxxxxxxxx,xx.xxx.xx.x,xx.xxx.xx.xx,xxxxx,xxx,T,O,A,E,,,,","xxxxxxxxxx,xx.xxx.xx.x,xx.xxx.x.x,xxxxx,xxx,T,O,A,E,,,,","xxxxxxxxxx,xx.xxx.xx.x,xx.xxx.x.x,xxxxx,xxx,T,O,A,E,,,,","xxxxxxxxxx,xx.xxx.xx.x,xx.xxx.x.x,xxxxx,xxx,T,O,A,B,,,,","xxxxxxxxxx,xx.xxx.xx.x,xx.xxx.x.xx,xxxxx,xxxx,T,O,A,B,,,,","xxxxxxxxxx,xx.xxx.xx.x,xx.xxx.x.x,xxxxx,xxx,T,O,A,B,,,,","xxxxxxxxxx,xx.xxx.xx.x,xx.xxx.x.x,xxxxx,xxx,T,O,A,E,,,,","xxxxxxxxxx,xx.xxx.xx.x,xx.xxx.xxx.xx,xxxxx,xxxx,T,O,A,E,,,,","xxxxxxxxxx,xx.xxx.xx.x,xx.xxx.x.x,xxxxx,xxx,T,O,A,E,,,,","xxxxxxxxxx,xx.xxx.xx.x,xx.xxx.x.x,xxxxx,xxx,T,O,A,E,,,,","xxxxxxxxxx,xx.xxx.xx.x,xx.xxx.x.x,xxxxx,xxx,T,O,A,B,,,,","xxxxxxxxxx,xx.xxx.xx.x,xx.xxx.x.x,xxxxx,xxx,T,O,A,E,,,,","xxxxxxxxxx,xx.xxx.xx.x,xx.xxx.x.x,xxxxx,xxx,T,O,A,E,,,,","xxxxxxxxxx,xx.xxx.xx.x,xx.xxx.x.x,xxxxx,xxx,T,O,A,B,,,,","xxxxxxxxxx,xx.xxx.xx.x,xx.xxx.xxx.xxx,xxxxx,xxxx,T,O,A,B,,,,","xxxxxxxxxx,xx.xxx.xx.x,xx.xxx.x.x,xxxxx,xxx,T,O,A,B,,,,","xxxxxxxxxx,xx.xxx.xx.x,xx.xxx.x.x,xxxxx,xxx,T,O,A,E,,,,","xxxxxxxxxx,xx.xxx.xx.x,xx.xxx.x.x,xxxxx,xxx,T,O,A,B,,,,","xxxxxxxxxx,xx.xxx.xx.x,x.xx.xx.xxx,xxxxx,xxxx,T,O,A,B,,,,","xxxxxxxxxx,xx.xxx.xx.x,xx.xxx.x.xx,xxxxx,xxxx,T,O,A,E,,,,","xxxxxxxxxx,xx.xxx.xx.x,xx.xxx.x.x,xxxxx,xxx,T,O,A,E,,,,"]}]}]}},{"time":"xxxx-xx-xxTxx:xx:xx.xxxxxxxZ","systemId":"bxxxexxx-xfxx-xbxe-becx-xxdfxxxxxxxx","macAddress":"xxxDxAxEExCx","category":"NetworkSecurityGroupFlowEvent","resourceId":"/SUBSCRIPTIONS/XXxXXXx-XXXx-xxxx-Xxxx-xXxxxxXXxX/RESOURCEGROUPS/MCSPUSxNETWORK/PROVIDERS/MICROSOFT.NETWORK/NETWORKSECURITYGROUPS/USxPORINDI","operationName":"NetworkSecurityGroupFlowEvents","properties":{"Version":x,"flows":[{"rule":"UserRule_servicechecks","flows":[{"mac":"xxxDxAxEExCx","flowTuples":["xxxxxxxxxx,xx.xxx.x.xxx,xx.xxx.xx.x,xxxxx,xxx,T,I,A,E,x,xxx,x,xxx","xxxxxxxxxx,xx.xxx.x.xxx,xx.xxx.xx.x,xxxxx,xxx,T,I,A,B,,,,","xxxxxxxxxx,xx.xxx.x.xxx,xx.xxx.xx.x,xxxxx,xxx,T,I,A,E,x,xxx,x,xxx","xxxxxxxxxx,xx.xxx.x.xxx,xx.xxx.xx.x,xxxxx,xxx,T,I,A,B,,,,","xxxxxxxxxx,xx.xxx.x.xxx,xx.xxx.xx.x,xxxxx,xxx,T,I,A,E,x,xxx,x,xxx","xxxxxxxxxx,xx.xxx.x.xxx,xx.xxx.xx.x,xxxxx,xxx,T,I,A,B,,,,","xxxxxxxxxx,xx.xxx.x.xxx,xx.xxx.xx.x,xxxxx,xxx,T,I,A,E,x,xxx,x,xxx","xxxxxxxxxx,xx.xxx.x.xxx,xx.xxx.xx.x,xxxxx,xxx,T,I,A,B,,,,","xxxxxxxxxx,xx.xxx.x.xxx,xx.xxx.xx.x,xxxxx,xxx,T,I,A,E,x,xxx,x,xxx","xxxxxxxxxx,xx.xxx.x.xxx,xx.xxx.xx.x,xxxxx,xxx,T,I,A,B,,,,","xxxxxxxxxx,xx.xxx.x.xxx,xx.xxx.xx.x,xxxxx,xxx,T,I,A,E,x,xxx,x,xxx","xxxxxxxxxx,xx.xxx.x.xxx,xx.xxx.xx.x,xxxxx,xxx,T,I,A,B,,,,","xxxxxxxxxx,xx.xxx.x.xxx,xx.xxx.xx.x,xxxxx,xxx,T,I,A,E,x,xxx,x,xxx","xxxxxxxxxx,xx.xxx.x.xxx,xx.xxx.xx.x,xxxxx,xxx,T,I,A,B,,,,","xxxxxxxxxx,xx.xxx.x.xxx,xx.xxx.xx.x,xxxxx,xxx,T,I,A,E,x,xxx,x,xxx","xxxxxxxxxx,xx.xxx.x.xxx,xx.xxx.xx.x,xxxxx,xxx,T,I,A,B,,,,","xxxxxxxxxx,xx.xxx.x.xxx,xx.xxx.xx.x,xxxxx,xxx,T,I,A,E,x,xxx,x,xxx"]}]},{"rule":"DefaultRule_AllowVnetOutBound","flows":[{"mac":"xxxSxAxSSxVx","flowTuples":["xxxxxxxxxx,xx.xxx.xx.x,xx.xxx.x.x,xxxxx,xxx,T,O,A,B,,,,","xxxxxxxxxx,xx.xxx.xx.x,xx.xxx.x.x,xxxxx,xxx,T,O,A,E,,,,","xxxxxxxxxx,xx.xxx.xx.x,xx.xxx.x.x,xxxxx,xxx,T,O,A,B,,,,","xxxxxxxxxx,xx.xxx.xx.x,xx.xxx.xx.xx,xxxxx,xxx,U,O,A,B,,,,","xxxxxxxxxx,xx.xxx.xx.x,xx.xxx.xx.xx,xxxxx,xxx,T,O,A,B,,,,","xxxxxxxxxx,xx.xxx.xx.x,xx.xxx.x.x,xxxxx,xxx,T,O,A,E,,,,","xxxxxxxxxx,xx.xxx.xx.x,xx.xxx.x.x,xxxxx,xxx,T,O,A,B,,,,","xxxxxxxxxx,xx.xxx.xx.x,xx.xxx.x.x,xxxxx,xxx,T,O,A,B,,,,","xxxxxxxxxx,xx.xxx.xx.x,xx.xxx.x.x,xxxxx,xxxxx,T,O,A,B,,,,","xxxxxxxxxx,xx.xxx.xx.x,xx.xxx.x.x,xxxxx,xxx,T,O,A,B,,,,","xxxxxxxxxx,xx.xxx.xx.x,xx.xxx.x.x,xxxxx,xxx,T,O,A,B,,,,","xxxxxxxxxx,xx.xxx.xx.x,xx.xxx.x.x,xxxxx,xxx,T,O,A,B,,,,","xxxxxxxxxx,xx.xxx.xx.x,xx.xxx.x.x,xxxxx,xxx,T,O,A,B,,,,","xxxxxxxxxx,xx.xxx.xx.x,xx.xxx.x.x,xxxxx,xxx,T,O,A,B,,,,","xxxxxxxxxx,xx.xxx.xx.x,xx.xxx.x.x,xxxxx,xxx,T,O,A,B,,,,","xxxxxxxxxx,xx.xxx.xx.x,xx.xxx.x.x,xxxxx,xxx,T,O,A,B,,,,","xxxxxxxxxx,xx.xxx.xx.x,xx.xxx.x.x,xxxxx,xxx,T,O,A,B,,,,","xxxxxxxxxx,xx.xxx.xx.x,xx.xxx.x.x,xxxxx,xxxxx,T,O,A,B,,,,","xxxxxxxxxx,xx.xxx.xx.x,xx.xxx.x.xx,xxxxx,xxxx,T,O,A,B,,,,","xxxxxxxxxx,xx.xxx.xx.x,xx.xxx.x.x,xxxxx,xxx,T,O,A,B,,,,","xxxxxxxxxx,xx.xxx.xx.x,x.xx.xx.xxx,xxxxx,xxxx,T,O,A,E,,,,","xxxxxxxxxx,xx.xxx.xx.x,xx.xxx.x.x,xxxxx,xxx,T,O,A,E,,,,","xxxxxxxxxx,xx.xxx.xx.x,xx.xxx.x.x,xxxxx,xxx,T,O,A,B,,,,","xxxxxxxxxx,xx.xxx.xx.x,xx.xxx.x.x,xxxxx,xxx,T,O,A,E,xxx,xxxxx,xxx,xxxxxx","xxxxxxxxxx,xx.xxx.xx.x,xx.xxx.x.x,xxxxx,xxx,T,O,A,E,x,xxx,x,xxx","xxxxxxxxxx,xx.xxx.xx.x,xx.xxx.x.x,xxxxx,xxx,T,O,A,B,,,,","xxxxxxxxxx,xx.xxx.xx.x,xx.xxx.x.x,xxxxx,xxx,T,O,A,E,,,,","xxxxxxxxxx,xx.xxx.xx.x,xx.xxx.x.x,xxxxx,xxx,T,O,A,B,,,,","xxxxxxxxxx,xx.xxx.xx.x,xx.xxx.x.x,xxxxx,xxx,T,O,A,E,xx,xxxx,xx,xxxxx","xxxxxxxxxx,xx.xxx.xx.x,xx.xxx.x.x,xxxxx,xxxxx,T,O,A,E,xx,xxxx,xx,xxxxx","xxxxxxxxxx,xx.xxx.xx.x,xx.xxx.x.x,xxxxx,xxx,T,O,A,E,xx,xxxxx,xx,xxxxx","xxxxxxxxxx,xx.xxx.xx.x,xx.xxx.x.x,xxxxx,xxx,T,O,A,E,xx,xxxx,xx,xxxxx","xxxxxxxxxx,xx.xxx.xx.x,xx.xxx.x.x,xxxxx,xxx,T,O,A,E,,,,","xxxxxxxxxx,xx.xxx.xx.x,xx.xxx.x.x,xxxxx,xxx,T,O,A,B,,,,","xxxxxxxxxx,xx.xxx.xx.x,xx.xxx.xxx.xxx,xxxxx,xxxx,T,O,A,B,,,,","xxxxxxxxxx,xx.xxx.xx.x,xx.xxx.x.x,xxxxx,xxx,T,O,A,E,,,,","xxxxxxxxxx,xx.xxx.xx.x,xx.xxx.x.x,xxxxx,xxx,T,O,A,B,,,,","xxxxxxxxxx,xx.xxx.xx.x,xx.xxx.x.x,xxxxx,xxx,T,O,A,E,,,,","xxxxxxxxxx,xx.xxx.xx.x,xx.x.xx.xxx,xxxxx,xxx,T,O,A,B,,,,","xxxxxxxxxx,xx.xxx.xx.x,xx.xxx.x.x,xxxxx,xxx,T,O,A,E,,,,","xxxxxxxxxx,xx.xxx.xx.x,xx.xxx.xxx.xx,xxxxx,xxxx,T,O,A,B,,,,","xxxxxxxxxx,xx.xxx.xx.x,xx.xxx.x.xx,xxxxx,xxxx,T,O,A,E,,,,","xxxxxxxxxx,xx.xxx.xx.x,xx.xxx.x.x,xxxxx,xxx,T,O,A,E,,,,"]}]}]}},{"time":"xxxx-xx-xxTxx:xx:xx.xxxxxxxZ","systemId":"bxxxexxx-xfxx-xbxe-becx-xxdfxxxxxxxx","macAddress":"xxxSxSxXXxCx","category":"NetworkSecurityGroupFlowEvent","resourceId":"/SUBSCRIPTIONS/XXxXXXx-XXXx-xxxx-Xxxx-xXxxxxXXxXxX/RESOURCEGROUPS/MCSPUSxNETWORK/PROVIDERS/MICROSOFT.NETWORK/NETWORKSECURITYGROUPS/USxXXXXXXX","operationName":"NetworkSecurityGroupFlowEvents","properties":{"Version":x,"flows":[{"rule":"UserRule_servicechecks","flows":[{"mac":"xxxDxAxEExCx","flowTuples":["xxxxxxxxxx,xx.xxx.x.xxx,xx.xxx.xx.x,xxxxx,xxx,T,I,A,B,,,,","xxxxxxxxxx,xx.xxx.x.xxx,xx.xxx.xx.x,xxxxx,xxx,T,I,A,E,x,xxx,x,xxx","xxxxxxxxxx,xx.xxx.x.xxx,xx.xxx.xx.x,xxxxx,xxx,T,I,A,B,,,,","xxxxxxxxxx,xx.xxx.x.xxx,xx.xxx.xx.x,xxxxx,xxx,T,I,A,E,x,xxx,x,xxx","xxxxxxxxxx,xx.xxx.x.xxx,xx.xxx.xx.x,xxxxx,xxx,T,I,A,B,,,,","xxxxxxxxxx,xx.xxx.x.xxx,xx.xxx.xx.x,xxxxx,xxx,T,I,A,E,x,xxx,x,xxx","xxxxxxxxxx,xx.xxx.x.xxx,xx.xxx.xx.x,xxxxx,xxx,T,I,A,B,,,,","xxxxxxxxxx,xx.xxx.x.xxx,xx.xxx.xx.x,xxxxx,xxx,T,I,A,E,x,xxx,x,xxx","xxxxxxxxxx,xx.xxx.x.xxx,xx.xxx.xx.x,xxxxx,xxx,T,I,A,B,,,,","xxxxxxxxxx,xx.xxx.x.xxx,xx.xxx.xx.x,xxxxx,xxx,T,I,A,E,x,xxx,x,xxx","xxxxxxxxxx,xx.xxx.x.xxx,xx.xxx.xx.x,xxxxx,xxx,T,I,A,B,,,,","xxxxxxxxxx,xx.xxx.x.xxx,xx.xxx.xx.x,xxxxx,xxx,T,I,A,E,x,xxx,x,xxx","xxxxxxxxxx,xx.xxx.x.xxx,xx.xxx.xx.x,xxxxx,xxx,T,I,A,B,,,,","xxxxxxxxxx,xx.xxx.x.xxx,xx.xxx.xx.x,xxxxx,xxx,T,I,A,E,x,xxx,x,xxx","xxxxxxxxxx,xx.xxx.x.xxx,xx.xxx.xx.x,xxxxx,xxx,T,I,A,B,,,,"]}]},

[anandhalagarasa]

Can anyone help please.

[anandhalagarasa]

Kindly help on the request.

[anandhalagarasa]

Hi anyone can you kindly help on my request please.