How to group different errors based on error types from the logs to create alert for continuous errors

[lladi]

I am creating a continuous error alert in Splunk. I have been working on constructing a search query to group different error types in Splunk. I have made several attempts and have explored multiple approaches; however, I have encountered challenges in effectively grouping the error types within the query.

Can anybody help me in this

[lladi]

 logger:integration-fabrics-exp-api.put:\orders\submit\(storeid).Exception
   message: [10-12 05:36:03] INFO Exception [[MuleRuntime].uber.12973: [integration-fabrics-exp-api-prod].util:logger-exception/processors/0.ps.BLOCKING @8989]: { "correlationId" : "787979-50ac-4b6f-90bd-64f1b6f79985", "message" : "Exception", "tracePoint" : "EXCEPTION", "priority" : "INFO", "category" : "kfc-integration-fabrics-exp-api.put:\\orders\\submit\\(storeid).Exception", "elapsed" : 3806, "locationInfo" : { "lineInFile" : "69", "component" : "json-logger:logger", "fileName" : "common/common-logger-flow.xml", "rootContainer" : "util:logger-exception" }, "timestamp" : "2023-10-12T05:36:03.317Z", "content" : { "payload" : { "api" : "integration-fabrics-exp-api-prod", "message" : "{\n \"externalOrderId\": \"275769403\",\n \"instruction\": \"275769403\",\n \"items\": [\n {\n \"id\": \"I-30995\",\n \"name\": \"Regular Chips\",\n \"unitPrice\": 445,\n \"quantity\": 1,\n \"subItems\": []\n },\n {\n \"id\": \"I-30057\",\n \"name\": \"Regular Potato \\u0026 Gravy\",\n \"unitPrice\": 545,\n \"quantity\": 1,\n \"subItems\": []\n },\n {\n \"id\": \"I-30017\",\n \"name\": \"3 Wicked Wings®\",\n \"unitPrice\": 695,\n \"quantity\": 1,\n \"subItems\": []\n },\n {\n \"id\": \"I-898-0\",\n \"name\": \"Kids Meal with Nuggets\",\n \"unitPrice\": 875,\n \"quantity\": 1,\n \"subItems\": [\n {\n \"id\": \"M-41687-0\",\n \"name\": \"4 Nuggets\",\n \"unitPrice\": 0,\n \"quantity\": 1,\n \"subItems\": []\n },\n {\n \"id\": \"M-40976-0\",\n \"name\": \"Regular Chips\",\n \"unitPrice\": 0,\n \"quantity\": 1,\n \"subItems\": []\n },\n {\n \"id\": \"M-40931-0\",\n \"name\": \"Regular 7Up\",\n \"unitPrice\": 0,\n \"quantity\": 1,\n \"subItems\": []\n }\n ]\n },\n {\n \"id\": \"I-32368-0\",\n \"name\": \"Kids Meal with Nuggets\",\n \"unitPrice\": 875,\n \"quantity\": 1,\n \"subItems\": [\n {\n \"id\": \"M-41687-0\",\n \"name\": \"4 Nuggets\",\n \"unitPrice\": 0,\n \"quantity\": 1,\n \"subItems\": []\n },\n {\n \"id\": \"M-40976-0\",\n \"name\": \"Regular Chips\",\n \"unitPrice\": 0,\n \"quantity\": 1,\n \"subItems\": []\n },\n {\n \"id\": \"M-40931-0\",\n \"name\": \"Regular 7Up\",\n \"unitPrice\": 0,\n \"quantity\": 1,\n \"subItems\": []\n }\n ]\n }\n ],\n \"customer\": {\n \"firstName\": \"9403\",\n \"lastName\": \"ML\",\n \"email\": \"ghgjhgj@hotmail.com\",\n \"phoneNumber\": \"897987\"\n },\n \"tenders\": [\n {\n \"type\": \"credit-card\",\n \"amount\": 3435\n }\n ],\n \"discountLines\": []\n}", "description" : "HTTP PUT on resource 'http://mule-worker-internal-order-sys-api-prod.au-s1.cloudhub.io:8091/orders/submit/716' failed: bad request (400).", "correlationId" : "1cb22ac0-50ac-4b6f-0988-64f1b6f79985", "category" : "integration-fabrics-exp-api.put:\\orders\\submit\\(storeid)", "timeStamp" : "2023-10-12T16:36:03:316000Z", "incomingMessage" : { "externalOrderId" : "9898", "instruction" : "275769403", "items" : [ { "id" : "I-30995", "name" : "Regular Chips", "unitPrice" : 445, "quantity" : 1, "subItems" : [ ] }, { "id" : "I-30057", "name" : "Regular Potato & Gravy", "unitPrice" : 545, "quantity" : 1, "subItems" : [ ] }, { "id" : "I-30017", "name" : "3 Wicked Wings®", "unitPrice" : 695, "quantity" : 1, "subItems" : [ ] }, { "id" : "I-32368-0", "name" : "Kids Meal with Nuggets", "unitPrice" : 875, "quantity" : 1, "subItems" : [ { "id" : "M-41687-0", "name" : "4 Nuggets", "unitPrice" : 0, "quantity" : 1, "subItems" : [ ] }, { "id" : "M-40976-0", "name" : "Regular Chips", "unitPrice" : 0, "quantity" : 1, "subItems" : [ ] }, { "id" : "M-40931-0", "name" : "Regular 7Up", "unitPrice" : 0, "quantity" : 1, "subItems" : [ ] } ] }, { "id" : "I-32368-0", "name" : "Kids Meal with Nuggets", "unitPrice" : 875, "quantity" : 1, "subItems" : [ { "id" : "M-41687-0", "name" : "4 Nuggets", "unitPrice" : 0, "quantity" : 1, "subItems" : [ ] }, { "id" : "M-40976-0", "name" : "Regular Chips", "unitPrice" : 0, "quantity" : 1, "subItems" : [ ] }, { "id" : "M-40931-0", "name" : "Regular 7Up", "unitPrice" : 0, "quantity" : 1, "subItems" : [ ] } ] } ], "customer" : { "firstName" : "9403", "lastName" : "ML", "email" : "ns@hotmail.com", "phoneNumber" : "98908" }, "tenders" : [ { "type" : "credit-card", "amount" : 3435 } ], "discountLines" : [ ] }, "errorMetadata" : { "errorType" : { "parentErrorType" : { "identifier" : "ANY", "namespace" : "MULE" }, "identifier" : "BAD_REQUEST", "namespace" : "HTTP" }, "description" : "HTTP PUT on resource 'http://mule-worker-internal-order-sys-api-prod.au-s1.cloudhub.io:8091/orders/submit/898' failed: bad request (400).", "additionalDetails" : "HTTP PUT on resource 'http://mule-worker-internal-order-sys-api-prod.au-s1.cloudhub.io:8091/orders/submit/716' failed: bad request (400).", "exception" : { "correlationId" : "1cb22ac0-50ac-4b6f-90bd-78979", "timestamp" : "2023-10-12T16:36:03:273000Z", "errorType" : "400 HTTP:BAD_REQUEST", "description" : "{\"code\":\"ghgj\",\"message\":\"CTT failed items. ModifierRequirementNotMet - 4 Nuggets,ModifierRequirementNotMet - 4 Nuggets\"}" } } } }, "applicationName" : "integration-fabrics-exp-api-prod", "applicationVersion" : "", "environment" : "prod", "threadName" : "[MuleRuntime].uber.12973: [integration-fabrics-exp-api-prod].util:logger-exception/processors/0.ps.BLOCKING @64c03d54" }

Here is the sample logger from which we need to group only the error message

, "description" : "{\"code\":\"ghgj\",\"message\":\"CTT failed items. ModifierRequirementNotMet - 4 Nuggets,ModifierRequirementNotMet - 4 Nuggets...and create alert checking if we aregetting more than 3 such continuous errors within an hour

[yuanliu]

In your explanations, the kind of error/exception messages you want to capture aren't completely consistent.  But I finally grasped that you want to count certain such messages based on actual message.  Note the sample data contain three "description" that indicate a failure:

content.payload.description	content.payload.errorMetadata.description	content.payload.errorMetadata.exception.description
HTTP PUT on resource http://mule-worker-internal-order-sys-api-prod.au-s1.cloudhub.io:8091/orders/submit/716 failed: bad request (400).	HTTP PUT on resource http://mule-worker-internal-order-sys-api-prod.au-s1.cloudhub.io:8091/orders/submit/898 failed: bad request (400).	{"code":"ghgj","message":"CTT failed items. ModifierRequirementNotMet - 4 Nuggets,ModifierRequirementNotMet - 4 Nuggets"}

I'll take content.payload.errorMetadata.exception.description as the one you want to capture most because this seems to be the closest match of what you said here.

But before that, you still need to explain more about your data.  If this illustration is the full raw log, it seems that the developer placed some plain text message in front of a conformant JSON object so the raw log is not JSON, even though the plain text looks like an excerpt from the JSON object itself, plus some meta data such as partial time stamp and log level.  To Splunk, this means that you do not directly get a field name content.payload.errorMetadata.exception.description.  Is this correct?  In other words, your "blocker" is not so much count by error, but how to extract that error message.

If this is the case, the problem is easily solved by removing the leading plain text and get to JSON, then use spath to extract fields from JSON.

| rex "[^{](?<json>{.+})"
| spath input=json
| stats count by content.payload.errorMetadata.exception.description

Your sample data will give

content.payload.errorMetadata.exception.description	count
{"code":"ghgj","message":"CTT failed items. ModifierRequirementNotMet - 4 Nuggets,ModifierRequirementNotMet - 4 Nuggets"}	1

(The count is 1 because there is only one sample) Is this something you are looking for?

Here is an emulation you can play with and compare with real data

| makeresults
| eval _raw = "logger:integration-fabrics-exp-api.put:\\orders\\submit\\(storeid).Exception
   message: [10-12 05:36:03] INFO Exception [[MuleRuntime].uber.12973: [integration-fabrics-exp-api-prod].util:logger-exception/processors/0.ps.BLOCKING @8989]: { \"correlationId\" : \"787979-50ac-4b6f-90bd-64f1b6f79985\", \"message\" : \"Exception\", \"tracePoint\" : \"EXCEPTION\", \"priority\" : \"INFO\", \"category\" : \"kfc-integration-fabrics-exp-api.put:\\\\orders\\\\submit\\\\(storeid).Exception\", \"elapsed\" : 3806, \"locationInfo\" : { \"lineInFile\" : \"69\", \"component\" : \"json-logger:logger\", \"fileName\" : \"common/common-logger-flow.xml\", \"rootContainer\" : \"util:logger-exception\" }, \"timestamp\" : \"2023-10-12T05:36:03.317Z\", \"content\" : { \"payload\" : { \"api\" : \"integration-fabrics-exp-api-prod\", \"message\" : \"{\\n \\\"externalOrderId\\\": \\\"275769403\\\",\\n \\\"instruction\\\": \\\"275769403\\\",\\n \\\"items\\\": [\\n {\\n \\\"id\\\": \\\"I-30995\\\",\\n \\\"name\\\": \\\"Regular Chips\\\",\\n \\\"unitPrice\\\": 445,\\n \\\"quantity\\\": 1,\\n \\\"subItems\\\": []\\n },\\n {\\n \\\"id\\\": \\\"I-30057\\\",\\n \\\"name\\\": \\\"Regular Potato \\\\u0026 Gravy\\\",\\n \\\"unitPrice\\\": 545,\\n \\\"quantity\\\": 1,\\n \\\"subItems\\\": []\\n },\\n {\\n \\\"id\\\": \\\"I-30017\\\",\\n \\\"name\\\": \\\"3 Wicked Wings®\\\",\\n \\\"unitPrice\\\": 695,\\n \\\"quantity\\\": 1,\\n \\\"subItems\\\": []\\n },\\n {\\n \\\"id\\\": \\\"I-898-0\\\",\\n \\\"name\\\": \\\"Kids Meal with Nuggets\\\",\\n \\\"unitPrice\\\": 875,\\n \\\"quantity\\\": 1,\\n \\\"subItems\\\": [\\n {\\n \\\"id\\\": \\\"M-41687-0\\\",\\n \\\"name\\\": \\\"4 Nuggets\\\",\\n \\\"unitPrice\\\": 0,\\n \\\"quantity\\\": 1,\\n \\\"subItems\\\": []\\n },\\n {\\n \\\"id\\\": \\\"M-40976-0\\\",\\n \\\"name\\\": \\\"Regular Chips\\\",\\n \\\"unitPrice\\\": 0,\\n \\\"quantity\\\": 1,\\n \\\"subItems\\\": []\\n },\\n {\\n \\\"id\\\": \\\"M-40931-0\\\",\\n \\\"name\\\": \\\"Regular 7Up\\\",\\n \\\"unitPrice\\\": 0,\\n \\\"quantity\\\": 1,\\n \\\"subItems\\\": []\\n }\\n ]\\n },\\n {\\n \\\"id\\\": \\\"I-32368-0\\\",\\n \\\"name\\\": \\\"Kids Meal with Nuggets\\\",\\n \\\"unitPrice\\\": 875,\\n \\\"quantity\\\": 1,\\n \\\"subItems\\\": [\\n {\\n \\\"id\\\": \\\"M-41687-0\\\",\\n \\\"name\\\": \\\"4 Nuggets\\\",\\n \\\"unitPrice\\\": 0,\\n \\\"quantity\\\": 1,\\n \\\"subItems\\\": []\\n },\\n {\\n \\\"id\\\": \\\"M-40976-0\\\",\\n \\\"name\\\": \\\"Regular Chips\\\",\\n \\\"unitPrice\\\": 0,\\n \\\"quantity\\\": 1,\\n \\\"subItems\\\": []\\n },\\n {\\n \\\"id\\\": \\\"M-40931-0\\\",\\n \\\"name\\\": \\\"Regular 7Up\\\",\\n \\\"unitPrice\\\": 0,\\n \\\"quantity\\\": 1,\\n \\\"subItems\\\": []\\n }\\n ]\\n }\\n ],\\n \\\"customer\\\": {\\n \\\"firstName\\\": \\\"9403\\\",\\n \\\"lastName\\\": \\\"ML\\\",\\n \\\"email\\\": \\\"ghgjhgj@hotmail.com\\\",\\n \\\"phoneNumber\\\": \\\"897987\\\"\\n },\\n \\\"tenders\\\": [\\n {\\n \\\"type\\\": \\\"credit-card\\\",\\n \\\"amount\\\": 3435\\n }\\n ],\\n \\\"discountLines\\\": []\\n}\", \"description\" : \"HTTP PUT on resource http://mule-worker-internal-order-sys-api-prod.au-s1.cloudhub.io:8091/orders/submit/716 failed: bad request (400).\", \"correlationId\" : \"1cb22ac0-50ac-4b6f-0988-64f1b6f79985\", \"category\" : \"integration-fabrics-exp-api.put:\\\\orders\\\\submit\\\\(storeid)\", \"timeStamp\" : \"2023-10-12T16:36:03:316000Z\", \"incomingMessage\" : { \"externalOrderId\" : \"9898\", \"instruction\" : \"275769403\", \"items\" : [ { \"id\" : \"I-30995\", \"name\" : \"Regular Chips\", \"unitPrice\" : 445, \"quantity\" : 1, \"subItems\" : [ ] }, { \"id\" : \"I-30057\", \"name\" : \"Regular Potato & Gravy\", \"unitPrice\" : 545, \"quantity\" : 1, \"subItems\" : [ ] }, { \"id\" : \"I-30017\", \"name\" : \"3 Wicked Wings®\", \"unitPrice\" : 695, \"quantity\" : 1, \"subItems\" : [ ] }, { \"id\" : \"I-32368-0\", \"name\" : \"Kids Meal with Nuggets\", \"unitPrice\" : 875, \"quantity\" : 1, \"subItems\" : [ { \"id\" : \"M-41687-0\", \"name\" : \"4 Nuggets\", \"unitPrice\" : 0, \"quantity\" : 1, \"subItems\" : [ ] }, { \"id\" : \"M-40976-0\", \"name\" : \"Regular Chips\", \"unitPrice\" : 0, \"quantity\" : 1, \"subItems\" : [ ] }, { \"id\" : \"M-40931-0\", \"name\" : \"Regular 7Up\", \"unitPrice\" : 0, \"quantity\" : 1, \"subItems\" : [ ] } ] }, { \"id\" : \"I-32368-0\", \"name\" : \"Kids Meal with Nuggets\", \"unitPrice\" : 875, \"quantity\" : 1, \"subItems\" : [ { \"id\" : \"M-41687-0\", \"name\" : \"4 Nuggets\", \"unitPrice\" : 0, \"quantity\" : 1, \"subItems\" : [ ] }, { \"id\" : \"M-40976-0\", \"name\" : \"Regular Chips\", \"unitPrice\" : 0, \"quantity\" : 1, \"subItems\" : [ ] }, { \"id\" : \"M-40931-0\", \"name\" : \"Regular 7Up\", \"unitPrice\" : 0, \"quantity\" : 1, \"subItems\" : [ ] } ] } ], \"customer\" : { \"firstName\" : \"9403\", \"lastName\" : \"ML\", \"email\" : \"ns@hotmail.com\", \"phoneNumber\" : \"98908\" }, \"tenders\" : [ { \"type\" : \"credit-card\", \"amount\" : 3435 } ], \"discountLines\" : [ ] }, \"errorMetadata\" : { \"errorType\" : { \"parentErrorType\" : { \"identifier\" : \"ANY\", \"namespace\" : \"MULE\" }, \"identifier\" : \"BAD_REQUEST\", \"namespace\" : \"HTTP\" }, \"description\" : \"HTTP PUT on resource http://mule-worker-internal-order-sys-api-prod.au-s1.cloudhub.io:8091/orders/submit/898 failed: bad request (400).\", \"additionalDetails\" : \"HTTP PUT on resource http://mule-worker-internal-order-sys-api-prod.au-s1.cloudhub.io:8091/orders/submit/716 failed: bad request (400).\", \"exception\" : { \"correlationId\" : \"1cb22ac0-50ac-4b6f-90bd-78979\", \"timestamp\" : \"2023-10-12T16:36:03:273000Z\", \"errorType\" : \"400 HTTP:BAD_REQUEST\", \"description\" : \"{\\\"code\\\":\\\"ghgj\\\",\\\"message\\\":\\\"CTT failed items. ModifierRequirementNotMet - 4 Nuggets,ModifierRequirementNotMet - 4 Nuggets\\\"}\" } } } }, \"applicationName\" : \"integration-fabrics-exp-api-prod\", \"applicationVersion\" : \"\", \"environment\" : \"prod\", \"threadName\" : \"[MuleRuntime].uber.12973: [integration-fabrics-exp-api-prod].util:logger-exception/processors/0.ps.BLOCKING @64c03d54\" }"
``` data emulation above ```

[yuanliu]

Illustrating raw data is an improvement.  Now could you describe the desired outcome, perhaps with a mock table, and describe the logic between the sample data and desired result?

[lladi]

I am getting blockers to group these different error messages from different exception events in our logs

[richgalloway]

Knowing what those blockers are would help us help you.  It would also help to know what search(es) you are running.

[lladi]

so like...table would be like :

Error Message: Object reference not set to an instance of an object.

description" : "{\"message\": \"Endpoint request timed out\

"message" : "EMERGENCY_SHUTDOWN || Products Not Available"

...such kind of different error messages with their number of occurences..

[richgalloway]

Yes, there probably are people here you can help you.  We do best, however, with specific questions rather than vague help requests.

Tell us what inputs you have and what results you'd like.  Show the failed attempts and say how they don't live up to expectations.  Describe the challenges you've encountered.

[lladi]

logger:integration-fabrics-exp-api.put:\orders\submit\(storeid).Exception
   message: [10-12 05:36:03] INFO Exception [[MuleRuntime].uber.12973: [integration-fabrics-exp-api-prod].util:logger-exception/processors/0.ps.BLOCKING @8989]: { "correlationId" : "787979-50ac-4b6f-90bd-64f1b6f79985", "message" : "Exception", "tracePoint" : "EXCEPTION", "priority" : "INFO", "category" : "kfc-integration-fabrics-exp-api.put:\\orders\\submit\\(storeid).Exception", "elapsed" : 3806, "locationInfo" : { "lineInFile" : "69", "component" : "json-logger:logger", "fileName" : "common/common-logger-flow.xml", "rootContainer" : "util:logger-exception" }, "timestamp" : "2023-10-12T05:36:03.317Z", "content" : { "payload" : { "api" : "integration-fabrics-exp-api-prod", "message" : "{\n \"externalOrderId\": \"275769403\",\n \"instruction\": \"275769403\",\n \"items\": [\n {\n \"id\": \"I-30995\",\n \"name\": \"Regular Chips\",\n \"unitPrice\": 445,\n \"quantity\": 1,\n \"subItems\": []\n },\n {\n \"id\": \"I-30057\",\n \"name\": \"Regular Potato \\u0026 Gravy\",\n \"unitPrice\": 545,\n \"quantity\": 1,\n \"subItems\": []\n },\n {\n \"id\": \"I-30017\",\n \"name\": \"3 Wicked Wings®\",\n \"unitPrice\": 695,\n \"quantity\": 1,\n \"subItems\": []\n },\n {\n \"id\": \"I-898-0\",\n \"name\": \"Kids Meal with Nuggets\",\n \"unitPrice\": 875,\n \"quantity\": 1,\n \"subItems\": [\n {\n \"id\": \"M-41687-0\",\n \"name\": \"4 Nuggets\",\n \"unitPrice\": 0,\n \"quantity\": 1,\n \"subItems\": []\n },\n {\n \"id\": \"M-40976-0\",\n \"name\": \"Regular Chips\",\n \"unitPrice\": 0,\n \"quantity\": 1,\n \"subItems\": []\n },\n {\n \"id\": \"M-40931-0\",\n \"name\": \"Regular 7Up\",\n \"unitPrice\": 0,\n \"quantity\": 1,\n \"subItems\": []\n }\n ]\n },\n {\n \"id\": \"I-32368-0\",\n \"name\": \"Kids Meal with Nuggets\",\n \"unitPrice\": 875,\n \"quantity\": 1,\n \"subItems\": [\n {\n \"id\": \"M-41687-0\",\n \"name\": \"4 Nuggets\",\n \"unitPrice\": 0,\n \"quantity\": 1,\n \"subItems\": []\n },\n {\n \"id\": \"M-40976-0\",\n \"name\": \"Regular Chips\",\n \"unitPrice\": 0,\n \"quantity\": 1,\n \"subItems\": []\n },\n {\n \"id\": \"M-40931-0\",\n \"name\": \"Regular 7Up\",\n \"unitPrice\": 0,\n \"quantity\": 1,\n \"subItems\": []\n }\n ]\n }\n ],\n \"customer\": {\n \"firstName\": \"9403\",\n \"lastName\": \"ML\",\n \"email\": \"ghgjhgj@hotmail.com\",\n \"phoneNumber\": \"897987\"\n },\n \"tenders\": [\n {\n \"type\": \"credit-card\",\n \"amount\": 3435\n }\n ],\n \"discountLines\": []\n}", "description" : "HTTP PUT on resource 'http://mule-worker-internal-order-sys-api-prod.au-s1.cloudhub.io:8091/orders/submit/716' failed: bad request (400).", "correlationId" : "1cb22ac0-50ac-4b6f-0988-64f1b6f79985", "category" : "integration-fabrics-exp-api.put:\\orders\\submit\\(storeid)", "timeStamp" : "2023-10-12T16:36:03:316000Z", "incomingMessage" : { "externalOrderId" : "9898", "instruction" : "275769403", "items" : [ { "id" : "I-30995", "name" : "Regular Chips", "unitPrice" : 445, "quantity" : 1, "subItems" : [ ] }, { "id" : "I-30057", "name" : "Regular Potato & Gravy", "unitPrice" : 545, "quantity" : 1, "subItems" : [ ] }, { "id" : "I-30017", "name" : "3 Wicked Wings®", "unitPrice" : 695, "quantity" : 1, "subItems" : [ ] }, { "id" : "I-32368-0", "name" : "Kids Meal with Nuggets", "unitPrice" : 875, "quantity" : 1, "subItems" : [ { "id" : "M-41687-0", "name" : "4 Nuggets", "unitPrice" : 0, "quantity" : 1, "subItems" : [ ] }, { "id" : "M-40976-0", "name" : "Regular Chips", "unitPrice" : 0, "quantity" : 1, "subItems" : [ ] }, { "id" : "M-40931-0", "name" : "Regular 7Up", "unitPrice" : 0, "quantity" : 1, "subItems" : [ ] } ] }, { "id" : "I-32368-0", "name" : "Kids Meal with Nuggets", "unitPrice" : 875, "quantity" : 1, "subItems" : [ { "id" : "M-41687-0", "name" : "4 Nuggets", "unitPrice" : 0, "quantity" : 1, "subItems" : [ ] }, { "id" : "M-40976-0", "name" : "Regular Chips", "unitPrice" : 0, "quantity" : 1, "subItems" : [ ] }, { "id" : "M-40931-0", "name" : "Regular 7Up", "unitPrice" : 0, "quantity" : 1, "subItems" : [ ] } ] } ], "customer" : { "firstName" : "9403", "lastName" : "ML", "email" : "ns@hotmail.com", "phoneNumber" : "98908" }, "tenders" : [ { "type" : "credit-card", "amount" : 3435 } ], "discountLines" : [ ] }, "errorMetadata" : { "errorType" : { "parentErrorType" : { "identifier" : "ANY", "namespace" : "MULE" }, "identifier" : "BAD_REQUEST", "namespace" : "HTTP" }, "description" : "HTTP PUT on resource 'http://mule-worker-internal-order-sys-api-prod.au-s1.cloudhub.io:8091/orders/submit/898' failed: bad request (400).", "additionalDetails" : "HTTP PUT on resource 'http://mule-worker-internal-order-sys-api-prod.au-s1.cloudhub.io:8091/orders/submit/716' failed: bad request (400).", "exception" : { "correlationId" : "1cb22ac0-50ac-4b6f-90bd-78979", "timestamp" : "2023-10-12T16:36:03:273000Z", "errorType" : "400 HTTP:BAD_REQUEST", "description" : "{\"code\":\"ghgj\",\"message\":\"CTT failed items. ModifierRequirementNotMet - 4 Nuggets,ModifierRequirementNotMet - 4 Nuggets\"}" } } } }, "applicationName" : "integration-fabrics-exp-api-prod", "applicationVersion" : "", "environment" : "prod", "threadName" : "[MuleRuntime].uber.12973: [integration-fabrics-exp-api-prod].util:logger-exception/processors/0.ps.BLOCKING @64c03d54" }

Here is the sample logger from which we need to group only the error message

, "description" : "{\"code\":\"ghgj\",\"message\":\"CTT failed items. ModifierRequirementNotMet - 4 Nuggets,ModifierRequirementNotMet - 4 Nuggets...and create alert checking if we aregetting more than 3 such continuous errors within an hour