topic Re: Trying to manually Create Regex for search in Splunk Search

Trying to manually Create Regex for search

sgoodman26 — Sat, 06 Oct 2018 04:43:58 GMT

We have been trying to create a search for AWS:Simple Email Services to locate any Bounce Back emails that come in; Specifically all Bounces and Complaints. We need the Fields created from the following points of data:

notificationType
emailAddress
diagnosticCode

We've tried using the Extract Fields, but since each data entry is formatted differently, it will only find the information on some, none on others. The issue we are running into is when trying to create a RegEx code manually; we've been using Reg101 which is able to pull the information that we need using these expressions:

diagnosticCode[^a-zA_Z0-9]*([^\\"]*)(?P<Diagnostic_Code>\\)

emailAddress[^a-zA_Z0-9]*(?P<ses_emailaddress>\\)

When plugging that into Splunk, that does not work, and pulls either random numbers or just . When looking at the code that is generated from the Extract Field generator, this is what comes up:

^(?:[^,\n]*,){10}\\\\\\"\w+\\\\\\":\\\\\\"(?P[^\\]+)

After reading all of the information in the About Splunk Regular Expressions document, I'm thinking that there is a different format that would be needed for the expression. One that is a bit more precise on where to look. Any help would be appreciative!

Re: Trying to manually Create Regex for search

Sukisen1981 — Sat, 06 Oct 2018 15:25:14 GMT

can you give some sample events?

Re: Trying to manually Create Regex for search

maciep — Sun, 07 Oct 2018 13:10:23 GMT

have you tried tried creating the extractions manually in the web instead of using the extract field generator? Settings -> Fields -> Field extractions -> New Field Extraction?

You should be able to use the same regex from 101. Also, maybe delete the ones created from the generator so there aren't any conflicts.

https://docs.splunk.com/Documentation/Splunk/7.1.3/Knowledge/Managesearch-timefieldextractions#Add_new_field_extractions_in_Splunk_Web

Re: Trying to manually Create Regex for search

sgoodman26 — Tue, 09 Oct 2018 16:26:45 GMT

Example:

{"MessageId": "28e14-4ed-42c-a3-f14816394", "Attributes": {"SentTimestamp": "1583", "SenderId": "AIDAIYLBIEIX46", "ApproximateReceiveCount": "1", "ApproximateFirstReceiveTimestamp": "152"}, "Body": "{\n \"Type\" : \"Notification\",\n \"MessageId\" : \"524-63-9-7-6096\",\n \"TopicArn\" : \"arn:aws:sns\",\n \"Message\" : \"{\\"notificationType\\":\\"Bounce\\",\\"bounce\\":{\\"bounceType\\":\\"Permanent\\",\\"bounceSubType\\":\\"General\\",\\"bouncedRecipients\\":[{\\"emailAddress\\":\\"emailaddress@domain.com\\",\\"action\\":\\"failed\\",\\"status\\":\\"5.1.1\\",\\"diagnosticCode\\":\\"smtp; 550 5.1.1 invalid recipient - Refer to Error Codes section at http://postmaster.cox.net/confluence/display/postmaster/Error+Codes for more information.\\"}],\\"timestamp\\":\\"2018\\",\\"feedbackId\\":\\"0101016640d3e5-a8c4-edf00\\",\\"remoteMtaIp\\":\\"1.1.1.1\\",\\"reportingMTA\\":\\"dsn; www.aws.com\\\"},\\\"mail\\\":{\\\"timestamp\\\":\\\"2018\\\",\\\"source\\\":\\\"no-reply@domain.com\\\",\\\"sourceArn\\\":\\\"arn:aws:ses:emailaddress@domain.com\\\",\\\"sourceIp\\\":\\\"1.1.1.1\\\",\\\"sendingAccountId\\\":\\\"111111111\\\",\\\"messageId\\\":\\\"010101010101\\\",\\\"destination\\\":[\\\"emailaddress@domain.com\\\"],

Re: Trying to manually Create Regex for search

493669 — Tue, 09 Oct 2018 17:06:55 GMT

try this :

    |rex "notificationType\\\\\\\":\\\\\\\"(?<notificationType>[^\\]+)"
|rex "emailAddress\\\\\\\":\\\\\\\"(?<emailAddress>[^\\]+)"
|rex "diagnosticCode\\\\\\\":\\\\\\\"(?<diagnosticCode>[^\\]+)"

https://regex101.com/r/Qm1nAu/1

also you can do search time extraction in props.conf file

Re: Trying to manually Create Regex for search

DalJeanis — Wed, 10 Oct 2018 16:30:14 GMT

Okay, one part of the problem here is your regex is either oversimplifying or overcomplicating the extraction ,depending on how you want to look at it.

Inside of this chunk...

[{\\\"emailAddress\\\":\\\"emailaddress@domain.com\\\",\\\"action\\\":\\\"failed\\\",\\\"status\\\":\\\"5.1.1\\\",\\\"diagnosticCode\\\":\\\"smtp; 550 5.1.1 invalid recipient - Refer to Error Codes section at http://postmaster.cox.net/confluence/display/postmaster/Error+Codes for more information.\\\"}]

...you want to find the words diagnosticCode, and if found, you want to extract everything that follows a set of multi-escaped quotes, a colon, and another set of multi-escaped quotes. In regex, that looks like this...

 diagnosticCode[\"\\\\]+:[\"\\\\]+(?<capturename>......some capture match goes here....)[\"\\\\]+

Now, each of those plusses probably has a minimum number of matches it's going to need to do to be proper. It looks like exactly 3 in the data you gave us, so it may be that each [\"\\\\]+ should be [\"\\\\]{3}.

Here's a run-anywhere example:

| makeresults 
| eval junk="\\"."\\"."\\\"" 
| eval _raw = "
[{".junk."emailAddress".junk.":".junk."emailaddress@domain.com".junk.",".junk."action".junk.":".junk."failed".junk.",".junk."status".junk.":".junk."5.1.1".junk.",".junk."diagnosticCode".junk.":".junk."smtp; 550 5.1.1 invalid recipient - Refer to Error Codes section at http://postmaster.cox.net/confluence/display/postmaster/Error+Codes for more information.".junk."}]"
| rex "diagnosticCode[\"\\\\]+:[\"\\\\]+(?<capturename>.+)(?<![\"\\\\])[\"\\\\]+"