Hello, I have been having trouble onboarding some logs that have some extra data at the top and are not breaking into individual events. I would like to remove the first 7 lines (I tried SEDCMD in props) and then break the following into individual events that start with "CEF:0". Any help would be appreciated. Sample log that came in as 1 event. accountId:1111111 configId:1111 checksum:fffffffffffffffffffffff format:CEF startTime:1591023419998 endTime:1591023786052 |==| CEF:0|Incapsula|SIEMintegration|1|1|Normal|0| fileId=763000040111111111 sourceServiceName=site.site.com siteid=41611111 suid=1111111 requestClientApplication=Mozilla/5.0 (compatible; MJ12bot/v1.4.8; http://mj12bot.com/) deviceFacility=ams cs2=false cs2Label=Javascript Support cs3=false cs3Label=Support cs1=NA cs1Label=Cap Support cs4=bf0e3ba9-cad7-42e3-917d-ffffffffffff cs4Label=VID cs5=a069314a28fc3f38df1a7fd08797ff70400c236c3f43c214a588d2c6b92fada93f21b37a01969be556f0370e4534fbd14969aa1f882f5680157c6c2cf9ffffff cs5Label=clappsig dproc=Unclassified cs6=Bot cs6Label=clapp ccode=DE cs7=51.2993 cs7Label=latitude cs8=9.491 cs8Label=longitude Customer=company start=1591023257044 request=site.site.com/products/ requestMethod=GET qstr=offset=70&max\ cn1=200 app=HTTPS act=REQ_PASSED deviceExternalId=107081971111111111 sip=x.x.x.x spt=443 in=6214 xff=x.x.x.x cpt=28286 src=x.x.x.x ver=TLSv1.2 ECDHE-RSA-AES128-GCM-SHA256 end=1591023257493 CEF:0|Incapsula|SIEMintegration|1|1|Normal|0| fileId=763000040111111111 sourceServiceName=site.site.com siteid=11111111 suid=1111111 requestClientApplication=Mozilla/5.0 (compatible; MJ12bot/v1.4.8; http://mj12bot.com/) deviceFacility=ams cs2=false cs2Label=Javascript Support cs3=false cs3Label=Support cs1=NA cs1Label=Cap Support cs4=bf0e3ba9-cad7-42e3-917d-ffffffffffff cs4Label=VID cs5=a069314a28fc3f38df1a7fd08797ff70400c236c3f43c214a588d2c6b92fada93f21b37a01969be556f0370e453411111111111f882f5680157c6c2cf9ac15cc cs5Label=clappsig dproc=Unclassified cs6=Bot cs6Label=clapp ccode=DE cs7=51.2993 cs7Label=latitude cs8=9.491 cs8Label=longitude Customer=company start=1591023260718 request=site.site.com/ requestMethod=GET qstr=offset=70&max cn1=302 app=HTTPS act=REQ_PASSED deviceExternalId=107082561111111111 sip=x.x.x.x spt=443 in=368 xff=x.x.x.x cpt=28286 src=x.x.x.x ver=TLSv1.2 ECDHE-RSA-AES128-GCM-SHA256 end=1591023260838 ... View more

Has anyone figured out how to extract the useful fields from Azure Firewall Logs? We are logging our Azure Firewall logs to a storage account and Splunk is pulling those with the Splunk Microsoft Cloud Services app. The app doesn't appear to include the field extractions for these types of logs. I am not well versed in regex, so I tried using the Splunk field extractor in the GUI, but I ran into issues. I mainly need the protocol, src_ip, src_port, dest_ip, dest_port, action. I was able to get this working for TCP and UDP traffic, but the problems started with ICMP traffic. My field extractions wouldn't match the ICMP traffic. I think this was due to no port field in those. Then I noticed a few other logs that would not be the same as the TCP/UDP logs either. Here are some sample scrubbed logs in case someone is willing to help here. Or if anyone has already solved this with some app that I am not aware of, I would appreciate some help! Thanks in advance. Here are all the sample logs examples I am seeing coming from the Azure Firewalls: { "category": "AzureFirewallNetworkRule", "time": "2019-07-30T17:43:59.9812590Z", "resourceId": "/SUBSCRIPTIONS/XXXX1111-XX11-XX11-XX11-XXXXXX111111/RESOURCEGROUPS/NET-XXX1-XXX1-RG/PROVIDERS/MICROSOFT.NETWORK/AZUREFIREWALLS/NET-XXX1-XXX1-FW01", "operationName": "AzureFirewallNetworkRuleLog", "properties": {"msg":"TCP request from XXX.X.178.18:42132 to XXX.X.242.12:51113. Action: Allow"}} { "category": "AzureFirewallNetworkRule", "time": "2019-07-30T17:44:59.4538600Z", "resourceId": "/SUBSCRIPTIONS/XXXX1111-XX11-XX11-XX11-XXXXXX111111/RESOURCEGROUPS/NET-XXX1-XXX1-RG/PROVIDERS/MICROSOFT.NETWORK/AZUREFIREWALLS/NET-XXX1-XXX1-FW02", "operationName": "AzureFirewallNetworkRuleLog", "properties": {"msg":"ICMP request from XXX.X.10.1 to XXX.X.69.5. Action: Allow"}} { "category": "AzureFirewallNetworkRule", "time": "2019-07-30T16:13:54.9901410Z", "resourceId": "/SUBSCRIPTIONS/XXXX1111-XX11-XX11-XX11-XXXXXX111111/RESOURCEGROUPS/DMZ-XXX1-XXX1-RG/PROVIDERS/MICROSOFT.NETWORK/AZUREFIREWALLS/DMZ-XXX1-XXX1-FW01", "operationName": "AzureFirewallNatRuleLog", "properties": {"msg":"TCP request from XXX.X.131.34:3318 to XXX.X.224.170:3299 was DNAT'ed to XXX.X.80.5:3299"}} { "category": "AzureFirewallNetworkRule", "time": "2019-07-30T16:39:45.5354460Z", "resourceId": "/SUBSCRIPTIONS/XXXX1111-XX11-XX11-XX11-XXXXXX111111/RESOURCEGROUPS/DMZ-XXX1-XXX1-RG/PROVIDERS/MICROSOFT.NETWORK/AZUREFIREWALLS/DMZ-XXX1-XXX1-FW01", "operationName": "AzureFirewallThreatIntelLog", "properties": {"msg":"TCP request from XXX.X.231.199:33348 to XXX.X.224.170:443. Action: Alert. ThreatIntel: Port Scan"}} { "category": "AzureFirewallApplicationRule", "time": "2019-07-30T17:19:00.1880780Z", "resourceId": "/SUBSCRIPTIONS/XXXX1111-XX11-XX11-XX11-XXXXXX111111/RESOURCEGROUPS/DMZ-XXX1-XXX1-RG/PROVIDERS/MICROSOFT.NETWORK/AZUREFIREWALLS/DMZ-XXX1-XXX1-FW01", "operationName": "AzureFirewallApplicationRuleLog", "properties": {"msg":"HTTPS request from XXX.X.110.8:65486 to abcd.efghijk.com:443. Action: Deny. No rule matched. Proceeding with default action"}} { "category": "AzureFirewallApplicationRule", "time": "2019-07-30T12:00:04.0868100Z", "resourceId": "/SUBSCRIPTIONS/XXXX1111-XX11-XX11-XX11-XXXXXX111111/RESOURCEGROUPS/DMZ-XXX1-XXX1-RG/PROVIDERS/MICROSOFT.NETWORK/AZUREFIREWALLS/DMZ-XXX1-XXX1-FW01", "operationName": "AzureFirewallApplicationRuleLog", "properties": {"msg":"HTTPS request from XXX.X.66.64:30476. Action: Deny. Reason: SNI TLS extension was missing."}} ... View more