topic Re: Possible to extract same value from different fields in props.conf? in Getting Data In

Possible to extract same value from different fields in props.conf?

jwalzerpitt — Tue, 09 Jul 2019 15:17:09 GMT

For Exchange message trace logs I am extracting the user as following in the props.conf file:

EXTRACT-user = "RecipientAddress":"(?<user>\S+)@

I would also like to extract the user from the SenderAddress as follows:

EXTRACT-user = "SenderAddress":"(?<user>\S+)@

Can I extract from two different fields for the user field, or will this potentially cause confusion?

Thx

Re: Possible to extract same value from different fields in props.conf?

FrankVl — Wed, 30 Sep 2020 01:12:20 GMT

It might make more sense to put one in src_user and the other in user. Or better yet, have a look at the definition for the CIM Email data model: https://docs.splunk.com/Documentation/CIM/4.13.0/User/Email and adhere to that.

If you really want to generate a multi-valued field user, with both sender and recipient(s), extract each into a separate field (e.g. s_user and r_user) and then do something like:

EVAL-user = mvappend(s_user,r_user)

Or use a REPORT extraction like this:

props.conf

REPORT-user = exchange-user

transforms.conf

[exchange-user]
REGEX = "(?:RecipientAddress|SenderAddress)":"(?<user>\S+)@
MV_ADD = true

Re: Possible to extract same value from different fields in props.conf?

jwalzerpitt — Tue, 09 Jul 2019 17:20:22 GMT

Frank,

Thx for the reply and info.

What I'm trying to do is leverage the message trace logs for a dashboard that is pulling from two other log sources, Duo and Shibboleth, that have 'user' defined as their user field, which is my main input field.

Would it make sense to make an alias for src_user in the message trace logs to 'user' so my dashboard would work across all three log sources?

Thx

Re: Possible to extract same value from different fields in props.conf?

woodcock — Tue, 09 Jul 2019 19:18:36 GMT

I would do it like this:

EXTRACT-user = "RecipientAddress":"(?<user_recipient>\S+)@
EXTRACT-user = "SenderAddress":"(?<user_sender>\S+)@
EVAL-user = mvappend(user_recipient, user_sender)

Re: Possible to extract same value from different fields in props.conf?

FrankVl — Wed, 10 Jul 2019 06:18:37 GMT

You can also resolve that in your dashboard query, by renaming the relevant fields in your search, to make them line up. How exactly to best do that depends on what exactly you want to achieve.

I'm not a big fan of putting very different data (sender and receiver) in 1 field, that is bound to lead to confusion. Especially since it is not compliant with the CIM Email data model.

Re: Possible to extract same value from different fields in props.conf?

jwalzerpitt — Thu, 11 Jul 2019 13:12:33 GMT

Thx for the feedback. I also am not a fan of changing the CIM email data model as well.

Re: Possible to extract same value from different fields in props.conf?

jwalzerpitt — Wed, 30 Sep 2020 01:17:03 GMT

Thx for the extracts.

My dashboard is currently set up to use 'user' as the value as my Shib and Duo logs have the user field. My message trace logs does not have the 'user' field, but 'src_user' instead. If I set the extracts you graciously provided, will this nullify the Email CIM data model of using 'src_user'?

Thx

Re: Possible to extract same value from different fields in props.conf?

woodcock — Thu, 11 Jul 2019 14:31:55 GMT

You are free to change EVAL-user to EVAL-src_user or EVAL-some_other_field_name and it will work that way. If you keep it as-is, it will override any existing value of user for this sourcetype.

If you would like to prefer any existing value, then use this:

 EVAL-user = coalesce(user, mvappend(user_recipient, user_sender))

If you would like to keep both any existing value and this new value, then use this:

EVAL-user = mvappend(user, user_recipient, user_sender)

This covers all possible desires. Be sure to click Accept to close the question.

Re: Possible to extract same value from different fields in props.conf?

jwalzerpitt — Thu, 11 Jul 2019 14:39:37 GMT

Thx for the various breakdowns