Extract Field from Multiple lines of 1 event

[rwiltzius2]

I have an event that I'm trying to extract the Email address between "Forwarding Address: " and ", Verification" The challenge is this event may have multiple Forwarding Addresses listed or none at all. I found a RegEx Testing site that I was able to create the following RegEx expression to match the three forwarding address below "(?<=Forwarding Address: )(.*)(?=,)", but it doesn't work within Splunk.

What is the best way that I can extract an unknown number of Forwarding Addresses for each event? Each Forwarding Address would be on a separate line, as shown below. Ideally, I would like to label any extraction that is between "Forwarding Address: " and "," as "Forwarding_Address". Please let me know if you have any questions and thank you in advance.

User: user1firstname.user1lastname@companydomain.com, Forwarding Addresses: (44/250)
Forwarding Address: user1firstname.user1lastname@domain1.com, Verification Status: accepted (1/3)
Forwarding Address: user2firstname.user2lastname@domain1.com, Verification Status: accepted (2/3)
Forwarding Address: distgroup@domain2.com, Verification Status: accepted (3/3)

[akocak]

In search time, (assuming event is multiline as in example):

| makeresults 
| eval _raw="User: user1firstname.user1lastname@companydomain.com, Forwarding Addresses: (44/250)
Forwarding Address: user1firstname.user1lastname@domain1.com, Verification Status: accepted (1/3)
Forwarding Address: user2firstname.user2lastname@domain1.com, Verification Status: accepted (2/3)
Forwarding Address: distgroup@domain2.com, Verification Status: accepted (3/3)" 
| rex mode=sed "s/\n/--BREAKER--/g" 
| eval raw_lines=split(_raw, "--BREAKER--") 
| mvexpand raw_lines
| rex field=raw_lines "\s(?<email>[^\s]+@[^\s]+)\,"
| fields -  _raw _time raw_lines

[martin_mueller · Accepted solution]

You're looking for the MV_ADD setting in transforms.conf to run an expression multiple times and have all matches added to a single multi-value field.

[martin_mueller]

Did you fix the case of REPORT-foo?

[rwiltzius2]

Thank you for having me verify. I overlooked it at first and just noticed it was still not in all uppercase. That did the trick!

[martin_mueller]

is GAM_Show_Forwarding_Address the sourcetype of your events?

Also, props.conf specs say REPORT-something in upper case.

[rwiltzius2]

Correct, GAM_Show_Forwarding_Address is the sourcetype of my events..

[martin_mueller]

Your regex looks like it has no named capturing group to actually define a field, try REGEX = Forwarding Address: (?<forwarding_address>[^,]+)

[rwiltzius2]

Thank you for your response. I've tried the following in transforms.conf:

[mv-type]
REGEX = (?<=Forwarding Address: )(?<gam_fa>[^,]+)
MV_ADD = true

and also:

[mv-type]
REGEX = Forwarding Address: (?<gam_fa>[^,]+)
MV_ADD = true

My props.conf is currently configured with:

[GAM_Show_Forwarding_Address]
Report-type = mv-type

Both files are located in /etc/system/local/.

I have been restarting Splunk after each change, but unfortunately, I do not see gam_fa listed as a field when I am searching. Please let me know if you have any questions.

[martin_mueller]

That feels wrong, and is probably changing the setting for nothing or everything.

Here's more human-friendly docs: http://docs.splunk.com/Documentation/Splunk/7.2.1/Knowledge/Exampleconfigurationsusingfieldtransform...

[rwiltzius2]

Thank you. So if the following regex expression, (?<=Forwarding Address: )(.*)(?=,), worked by identifying the multiple values in an event. Then I assume that in the transforms.conf I would add the following:

[mv-type]
REGEX = type=(?<=Forwarding Address: )(.*)(?=,)
MV_ADD = true

and then for props.conf:
REPORT-type = mv-type

Would you concur or is my Regex incorrect?

[martin_mueller]

Wait, what? Where did you update MV_ADD, under what stanza?

Also, read http://docs.splunk.com/Documentation/Splunk/7.2.1/Knowledge/Configureadvancedextractionswithfieldtra...

[rwiltzius2]

Sorry for my delayed response. From what I gather, I didn't update it under any particular stanza. I made the MV_ADD modification at the very top of the file.

I reviewed the link you provided and to be honest, I feel a bit overwhelmed. If you're willing to continue assisting me, I would greatly appreciate it. What information do you need from me to further troubleshoot this?

[mstjohn_splunk]

hi @rwiltzius2

Did you get a chance to look over martin_mueller's question? If you answer him back, he might be able to solve your query!

Thanks for posting!

[rwiltzius2]

Thank you. I copied the transforms.conf from /etc/system/default/ to /etc/system/local/. Then updated the MV_ADD to True and rebooted Splunk.

However, I still can't seem to create a proper RegEx field extraction to capture each Email address that is listed as a Forwarding Address on the multiple lines within one event.

Is this something you are able to assist with? Any help would be greatly appreciated.