Splunk Search

Trying to manually Create Regex for search

sgoodman26
Explorer

We have been trying to create a search for AWS:Simple Email Services to locate any Bounce Back emails that come in; Specifically all Bounces and Complaints. We need the Fields created from the following points of data:

notificationType
emailAddress
diagnosticCode

We've tried using the Extract Fields, but since each data entry is formatted differently, it will only find the information on some, none on others. The issue we are running into is when trying to create a RegEx code manually; we've been using Reg101 which is able to pull the information that we need using these expressions:

diagnosticCode[^a-zA_Z0-9]*([^\\"]*)(?P<Diagnostic_Code>\\)

emailAddress[^a-zA_Z0-9]*(?P<ses_emailaddress>\\) 

When plugging that into Splunk, that does not work, and pulls either random numbers or just . When looking at the code that is generated from the Extract Field generator, this is what comes up:

^(?:[^,\n]*,){10}\\\\\\"\w+\\\\\\":\\\\\\"(?P[^\\]+)

After reading all of the information in the About Splunk Regular Expressions document, I'm thinking that there is a different format that would be needed for the expression. One that is a bit more precise on where to look. Any help would be appreciative!

0 Karma
1 Solution

DalJeanis
Legend

Okay, one part of the problem here is your regex is either oversimplifying or overcomplicating the extraction ,depending on how you want to look at it.

Inside of this chunk...

[{\\\"emailAddress\\\":\\\"emailaddress@domain.com\\\",\\\"action\\\":\\\"failed\\\",\\\"status\\\":\\\"5.1.1\\\",\\\"diagnosticCode\\\":\\\"smtp; 550 5.1.1 invalid recipient - Refer to Error Codes section at http://postmaster.cox.net/confluence/display/postmaster/Error+Codes for more information.\\\"}]

...you want to find the words diagnosticCode, and if found, you want to extract everything that follows a set of multi-escaped quotes, a colon, and another set of multi-escaped quotes. In regex, that looks like this...

 diagnosticCode[\"\\\\]+:[\"\\\\]+(?<capturename>......some capture match goes here....)[\"\\\\]+

Now, each of those plusses probably has a minimum number of matches it's going to need to do to be proper. It looks like exactly 3 in the data you gave us, so it may be that each [\"\\\\]+ should be [\"\\\\]{3}.

Here's a run-anywhere example:

| makeresults 
| eval junk="\\"."\\"."\\\"" 
| eval _raw = "
[{".junk."emailAddress".junk.":".junk."emailaddress@domain.com".junk.",".junk."action".junk.":".junk."failed".junk.",".junk."status".junk.":".junk."5.1.1".junk.",".junk."diagnosticCode".junk.":".junk."smtp; 550 5.1.1 invalid recipient - Refer to Error Codes section at http://postmaster.cox.net/confluence/display/postmaster/Error+Codes for more information.".junk."}]"
| rex "diagnosticCode[\"\\\\]+:[\"\\\\]+(?<capturename>.+)(?<![\"\\\\])[\"\\\\]+"

View solution in original post

DalJeanis
Legend

Okay, one part of the problem here is your regex is either oversimplifying or overcomplicating the extraction ,depending on how you want to look at it.

Inside of this chunk...

[{\\\"emailAddress\\\":\\\"emailaddress@domain.com\\\",\\\"action\\\":\\\"failed\\\",\\\"status\\\":\\\"5.1.1\\\",\\\"diagnosticCode\\\":\\\"smtp; 550 5.1.1 invalid recipient - Refer to Error Codes section at http://postmaster.cox.net/confluence/display/postmaster/Error+Codes for more information.\\\"}]

...you want to find the words diagnosticCode, and if found, you want to extract everything that follows a set of multi-escaped quotes, a colon, and another set of multi-escaped quotes. In regex, that looks like this...

 diagnosticCode[\"\\\\]+:[\"\\\\]+(?<capturename>......some capture match goes here....)[\"\\\\]+

Now, each of those plusses probably has a minimum number of matches it's going to need to do to be proper. It looks like exactly 3 in the data you gave us, so it may be that each [\"\\\\]+ should be [\"\\\\]{3}.

Here's a run-anywhere example:

| makeresults 
| eval junk="\\"."\\"."\\\"" 
| eval _raw = "
[{".junk."emailAddress".junk.":".junk."emailaddress@domain.com".junk.",".junk."action".junk.":".junk."failed".junk.",".junk."status".junk.":".junk."5.1.1".junk.",".junk."diagnosticCode".junk.":".junk."smtp; 550 5.1.1 invalid recipient - Refer to Error Codes section at http://postmaster.cox.net/confluence/display/postmaster/Error+Codes for more information.".junk."}]"
| rex "diagnosticCode[\"\\\\]+:[\"\\\\]+(?<capturename>.+)(?<![\"\\\\])[\"\\\\]+"

maciep
Champion

have you tried tried creating the extractions manually in the web instead of using the extract field generator? Settings -> Fields -> Field extractions -> New Field Extraction?

You should be able to use the same regex from 101. Also, maybe delete the ones created from the generator so there aren't any conflicts.

https://docs.splunk.com/Documentation/Splunk/7.1.3/Knowledge/Managesearch-timefieldextractions#Add_n...

0 Karma

493669
Super Champion

try this :

    |rex "notificationType\\\\\\\":\\\\\\\"(?<notificationType>[^\\]+)"
|rex "emailAddress\\\\\\\":\\\\\\\"(?<emailAddress>[^\\]+)"
|rex "diagnosticCode\\\\\\\":\\\\\\\"(?<diagnosticCode>[^\\]+)"

https://regex101.com/r/Qm1nAu/1

also you can do search time extraction in props.conf file

0 Karma

Sukisen1981
Champion

can you give some sample events?

0 Karma

sgoodman26
Explorer

Example:

{"MessageId": "28e14-4ed-42c-a3-f14816394", "Attributes": {"SentTimestamp": "1583", "SenderId": "AIDAIYLBIEIX46", "ApproximateReceiveCount": "1", "ApproximateFirstReceiveTimestamp": "152"}, "Body": "{\n \"Type\" : \"Notification\",\n \"MessageId\" : \"524-63-9-7-6096\",\n \"TopicArn\" : \"arn:aws:sns\",\n \"Message\" : \"{\\"notificationType\\":\\"Bounce\\",\\"bounce\\":{\\"bounceType\\":\\"Permanent\\",\\"bounceSubType\\":\\"General\\",\\"bouncedRecipients\\":[{\\"emailAddress\\":\\"emailaddress@domain.com\\",\\"action\\":\\"failed\\",\\"status\\":\\"5.1.1\\",\\"diagnosticCode\\":\\"smtp; 550 5.1.1 invalid recipient - Refer to Error Codes section at http://postmaster.cox.net/confluence/display/postmaster/Error+Codes for more information.\\"}],\\"timestamp\\":\\"2018\\",\\"feedbackId\\":\\"0101016640d3e5-a8c4-edf00\\",\\"remoteMtaIp\\":\\"1.1.1.1\\",\\"reportingMTA\\":\\"dsn; www.aws.com\\\"},\\\"mail\\\":{\\\"timestamp\\\":\\\"2018\\\",\\\"source\\\":\\\"no-reply@domain.com...,

0 Karma
Get Updates on the Splunk Community!

New Release | Splunk Enterprise 9.3

Admins and Analyst can benefit from:  Seamlessly route data to your local file system to save on storage ...

2024 Splunk Career Impact Survey | Earn a $20 gift card for participating!

Hear ye, hear ye! The time has come again for Splunk's annual Career Impact Survey!  We need your help by ...

Optimize Cloud Monitoring

  TECH TALKS Optimize Cloud Monitoring Tuesday, August 13, 2024  |  11:00AM–12:00PM PST   Register to ...