How to line break events

[anasamer]

Can anyone here help with breaking this sample into multiple events each should start with { "resourceId": ?
I have the below log sample:

{"records": [{ "resourceId": "/SUBSCRIPTIONS/9799XXX5-F9BF-XXXX-XXXX-6DDXXXXXF4D99/RESOURCEGROUPS/DEVELOPMENT/PROVIDERS/MICROSOFT.NETWORK/APPLICATIONGATEWAYS/SPLUNKWAF", "operationName": "ApplicationGatewayFirewall", "time": "2019-06-11T11:09:39.2282087Z", "category": "ApplicationGatewayFirewallLog", "properties": {
  "instanceId": "ApplicationGatewayRole_IN_1",
  "clientIp": "10.0.1.5",
  "clientPort": "0",
  "requestUri": "/en-GB/splunkd/__raw/services/server/health/splunkd?output_mode=json&_=1560250716771",
  "ruleSetType": "OWASP",
  "ruleSetVersion": "3.0",
  "ruleId": "920350",
  "ruleGroup": "920-PROTOCOL-ENFORCEMENT",
  "message": "Host header is a numeric IP address",
  "action": "Matched",
  "site": "Global",
  "details": {
    "message": "Warning. Pattern match \"^[\\\\d.:]+$\" at REQUEST_HEADERS:Host.",
    "data": "10.10.10.10:8001",
    "file": "rules/REQUEST-920-PROTOCOL-ENFORCEMENT.conf",
    "line": "791"
  },
  "hostname": "10.10.10.10",
  "transactionId": "17726168135477758612"
}},{ "resourceId": "/SUBSCRIPTIONS/9799XXX5-F9BF-XXXX-XXXX-6DDXXXXXF4D99/RESOURCEGROUPS/DEVELOPMENT/PROVIDERS/MICROSOFT.NETWORK/APPLICATIONGATEWAYS/SPLUNKWAF", "operationName": "ApplicationGatewayFirewall", "time": "2019-06-11T11:09:43.2069335Z", "category": "ApplicationGatewayFirewallLog", "properties": {
  "instanceId": "ApplicationGatewayRole_IN_1",
  "clientIp": "10.0.1.5",
  "clientPort": "0",
  "requestUri": "/en-GB/splunkd/__raw/services/messages?output_mode=json&sort_key=timeCreated_epochSecs&sort_dir=desc&count=1000&_=1560250720227",
  "ruleSetType": "OWASP",
  "ruleSetVersion": "3.0",
  "ruleId": "920350",
  "ruleGroup": "920-PROTOCOL-ENFORCEMENT",
  "message": "Host header is a numeric IP address",
  "action": "Matched",
  "site": "Global",
  "details": {
    "message": "Warning. Pattern match \"^[\\\\d.:]+$\" at REQUEST_HEADERS:Host.",
    "data": "10.10.10.10:8001",
    "file": "rules/REQUEST-920-PROTOCOL-ENFORCEMENT.conf",
    "line": "791"
  },
  "hostname": "10.10.10.10",
  "transactionId": "17726168135477758613"
}},{ "resourceId": "/SUBSCRIPTIONS/9799XXX5-F9BF-XXXX-XXXX-6DDXXXXXF4D99/RESOURCEGROUPS/DEVELOPMENT/PROVIDERS/MICROSOFT.NETWORK/APPLICATIONGATEWAYS/SPLUNKWAF", "operationName": "ApplicationGatewayFirewall", "time": "2019-06-11T11:09:49.9545793Z", "category": "ApplicationGatewayFirewallLog", "properties": {
  "instanceId": "ApplicationGatewayRole_IN_1",
  "clientIp": "10.0.1.5",
  "clientPort": "0",
  "requestUri": "/en-GB/splunkd/__raw/services/messages?output_mode=json&sort_key=timeCreated_epochSecs&sort_dir=desc&count=1000&_=1560250716774",
  "ruleSetType": "OWASP",
  "ruleSetVersion": "3.0",
  "ruleId": "920350",
  "ruleGroup": "920-PROTOCOL-ENFORCEMENT",
  "message": "Host header is a numeric IP address",
  "action": "Matched",
  "site": "Global",
  "details": {
    "message": "Warning. Pattern match \"^[\\\\d.:]+$\" at REQUEST_HEADERS:Host.",
    "data": "10.10.10.10:8001",
    "file": "rules/REQUEST-920-PROTOCOL-ENFORCEMENT.conf",
    "line": "791"
  },
  "hostname": "10.10.10.10",
  "transactionId": "17726168135477758614"
}}]}
{"records": [{ "resourceId": "/SUBSCRIPTIONS/9799XXX5-F9BF-XXXX-XXXX-6DDXXXXXF4D99/RESOURCEGROUPS/DEVELOPMENT/PROVIDERS/MICROSOFT.NETWORK/APPLICATIONGATEWAYS/SPLUNKWAF", "operationName": "ApplicationGatewayFirewall", "time": "2019-06-11T11:08:59.1006429Z", "category": "ApplicationGatewayFirewallLog", "properties": {
  "instanceId": "ApplicationGatewayRole_IN_0",
  "clientIp": "10.0.1.7",
  "clientPort": "0",
  "requestUri": "/en-GB/splunkd/__raw/services/server/health/splunkd?output_mode=json&_=1560250716762",
  "ruleSetType": "OWASP",
  "ruleSetVersion": "3.0",
  "ruleId": "920350",
  "ruleGroup": "920-PROTOCOL-ENFORCEMENT",
  "message": "Host header is a numeric IP address",
  "action": "Matched",
  "site": "Global",
  "details": {
    "message": "Warning. Pattern match \"^[\\\\d.:]+$\" at REQUEST_HEADERS:Host.",
    "data": "10.10.10.10:8001",
    "file": "rules/REQUEST-920-PROTOCOL-ENFORCEMENT.conf",
    "line": "791"
  },
  "hostname": "10.10.10.10",
  "transactionId": "17365880165288120552"
}}]}

[richgalloway]

Have you tried LINE_BREAKER = ()\{ "resourceId": ?

[anasamer]

nope it is not working

[FrankVl]

Your image is not publicly visible. Make sure to fix the typo (you need capital I instead of lowercase).

[FrankVl]

The i in resourceId must be a capital I 🙂

Of course to be combined with SHOULD_LINEMERGE = false.

And a bit more specific linebreak to try could be: LINE_BREAKER = ((?:\]\})?[\r\n\s]*\{"records":\s\[|,)\{\s"resourceId":
That also strips out the , in between events and the records [ bit.

[richgalloway]

I fixed the 'I'.

[anasamer]

thanks @FrankVl this regex captured the records in the middle of the log but not capturing the first records

[anasamer]

regexr.com/4flnp

[FrankVl]

That is not the same regex I shared 😉

You added a closing } in between the character class that is matching linebreaks and the * behind it. And then indeed it doesn't work properly. Your fix is also incorrect, as it should be a * to add, not a ?, since there could be a combination of multiple linebreak and whitespace characters.

But you simply shouldn't add that } there in the first place, as it will result in stripping that off from the end of the previous event and that will break your json syntax.

I think it should work if you use the exact regex I shared, but do let me know if it doesn't: https://regexr.com/4flu5

[anasamer]

I fixed it by adding ? so it will be like

((?:\]\})*[\r\n\s]?}*\{"records":\s\[|,)\{\s*"resourceId":

Thanks