Trying to manually Create Regex for search

[sgoodman26]

We have been trying to create a search for AWS:Simple Email Services to locate any Bounce Back emails that come in; Specifically all Bounces and Complaints. We need the Fields created from the following points of data:

notificationType
emailAddress
diagnosticCode

We've tried using the Extract Fields, but since each data entry is formatted differently, it will only find the information on some, none on others. The issue we are running into is when trying to create a RegEx code manually; we've been using Reg101 which is able to pull the information that we need using these expressions:

diagnosticCode[^a-zA_Z0-9]*([^\\"]*)(?P<Diagnostic_Code>\\)

emailAddress[^a-zA_Z0-9]*(?P<ses_emailaddress>\\) 

When plugging that into Splunk, that does not work, and pulls either random numbers or just . When looking at the code that is generated from the Extract Field generator, this is what comes up:

^(?:[^,\n]*,){10}\\\\\\"\w+\\\\\\":\\\\\\"(?P[^\\]+)

After reading all of the information in the About Splunk Regular Expressions document, I'm thinking that there is a different format that would be needed for the expression. One that is a bit more precise on where to look. Any help would be appreciative!

[DalJeanis · Accepted solution]

Okay, one part of the problem here is your regex is either oversimplifying or overcomplicating the extraction ,depending on how you want to look at it.

Inside of this chunk...

[{\\\"emailAddress\\\":\\\"[email protected]\\\",\\\"action\\\":\\\"failed\\\",\\\"status\\\":\\\"5.1.1\\\",\\\"diagnosticCode\\\":\\\"smtp; 550 5.1.1 invalid recipient - Refer to Error Codes section at http://postmaster.cox.net/confluence/display/postmaster/Error+Codes for more information.\\\"}]

...you want to find the words diagnosticCode, and if found, you want to extract everything that follows a set of multi-escaped quotes, a colon, and another set of multi-escaped quotes. In regex, that looks like this...

 diagnosticCode[\"\\\\]+:[\"\\\\]+(?<capturename>......some capture match goes here....)[\"\\\\]+

Now, each of those plusses probably has a minimum number of matches it's going to need to do to be proper. It looks like exactly 3 in the data you gave us, so it may be that each [\"\\\\]+ should be [\"\\\\]{3}.

Here's a run-anywhere example:

| makeresults 
| eval junk="\\"."\\"."\\\"" 
| eval _raw = "
[{".junk."emailAddress".junk.":".junk."[email protected]".junk.",".junk."action".junk.":".junk."failed".junk.",".junk."status".junk.":".junk."5.1.1".junk.",".junk."diagnosticCode".junk.":".junk."smtp; 550 5.1.1 invalid recipient - Refer to Error Codes section at http://postmaster.cox.net/confluence/display/postmaster/Error+Codes for more information.".junk."}]"
| rex "diagnosticCode[\"\\\\]+:[\"\\\\]+(?<capturename>.+)(?<![\"\\\\])[\"\\\\]+"

[maciep]

have you tried tried creating the extractions manually in the web instead of using the extract field generator? Settings -> Fields -> Field extractions -> New Field Extraction?

You should be able to use the same regex from 101. Also, maybe delete the ones created from the generator so there aren't any conflicts.

https://docs.splunk.com/Documentation/Splunk/7.1.3/Knowledge/Managesearch-timefieldextractions#Add_n...

[493669]

try this :

    |rex "notificationType\\\\\\\":\\\\\\\"(?<notificationType>[^\\]+)"
|rex "emailAddress\\\\\\\":\\\\\\\"(?<emailAddress>[^\\]+)"
|rex "diagnosticCode\\\\\\\":\\\\\\\"(?<diagnosticCode>[^\\]+)"

https://regex101.com/r/Qm1nAu/1

also you can do search time extraction in props.conf file

[Sukisen1981]

can you give some sample events?

[sgoodman26]

Example:

{"MessageId": "28e14-4ed-42c-a3-f14816394", "Attributes": {"SentTimestamp": "1583", "SenderId": "AIDAIYLBIEIX46", "ApproximateReceiveCount": "1", "ApproximateFirstReceiveTimestamp": "152"}, "Body": "{\n \"Type\" : \"Notification\",\n \"MessageId\" : \"524-63-9-7-6096\",\n \"TopicArn\" : \"arn:aws:sns\",\n \"Message\" : \"{\\"notificationType\\":\\"Bounce\\",\\"bounce\\":{\\"bounceType\\":\\"Permanent\\",\\"bounceSubType\\":\\"General\\",\\"bouncedRecipients\\":[{\\"emailAddress\\":\\"[email protected]\\",\\"action\\":\\"failed\\",\\"status\\":\\"5.1.1\\",\\"diagnosticCode\\":\\"smtp; 550 5.1.1 invalid recipient - Refer to Error Codes section at http://postmaster.cox.net/confluence/display/postmaster/Error+Codes for more information.\\"}],\\"timestamp\\":\\"2018\\",\\"feedbackId\\":\\"0101016640d3e5-a8c4-edf00\\",\\"remoteMtaIp\\":\\"1.1.1.1\\",\\"reportingMTA\\":\\"dsn; www.aws.com\\\"},\\\"mail\\\":{\\\"timestamp\\\":\\\"2018\\\",\\\"source\\\":\\\"[email protected]...,