topic Re: Regex with multiple fields duplicated. Yet data can be different? in Splunk Search

Regex with multiple fields duplicated. Yet data can be different?

dmacgillivray — Mon, 28 Sep 2020 16:35:42 GMT

I have an issue with data titles that would appear to be repeated, yet in the case below, The passwordexpiry_date: field can be repeated with different values.

See below account fields are replicated and I am trying to determine if a REGEX can even be written on this. Especially when field names are duplicated.

sequence_number=18112355,
remote_client=Eserver,
2014-05-08 18:02:47:84 GMT,
conn=72276,
op=239,
eventID=0a46b98715e113924708169294656,
messageID=701,
ip=127.0.0.1,
uname=gaspm05,
urole=Bobs And Admin,
msg=Modify user,
msgtype=MODIFY,
result=0,
etime=34ms,
upd_user_owner=Person User Admin Group,
associated_parent_member_groups=[ ],
upd_user=DATA_BOB_Truck3bobsurunc,
new_user=[userid:
[DATA_BOB_Truck3bobsurunc]
firstname:[Truck3bobsurunc]
lastname:[LN]
email:[]
certdn:[]
properties:[[name=ContactID isset=false], [name=ctscPasswordResetAttempts isset=false], [name=ctscSecretQuestionAnswer isset=false]]
account_start_date:[Mar 12, 2014 6:08:47 PM]
account_expiry_date:[Jan 1, 4712 6:08:47 PM]
islockedout:[false]
isactive:[true]
passwordexpiry_date:[Aug 6, 2014 6:02:47 PM]
admin_group:[Person User Admin Group]
ispublic:[true]],
original_user=[userid:[DATA_BOB_Truck3bobsurunc]
firstname:[Truck3bobsurunc]
lastname:[LN]
email:[]
certdn:[]
properties:[[name=ContactID isset=false], [name=ctscPasswordResetAttempts isset=false], [name=ctscSecretQuestionAnswer isset=false]]
account_start_date:[Mar 12, 2014 6:08:47 PM]
account_expiry_date:[Jan 1, 4712 6:08:47 PM]
islockedout:[false]
isactive:[true]
passwordexpiry_date:[Jul 30, 2014 6:26:37 PM]
admin_group:[Person User Admin Group]
ispublic:[true]],
user_owner=Person User Admin Group,
user=DATA_BOB_Truck3bobsurunc,
userUID=CCCCCQEBBBBDU1FMJJJJBlRlc3R2MgAAAAEQDDDDDDDkjBE=

Re: Regex with multiple fields duplicated. Yet data can be different?

richgalloway — Tue, 13 May 2014 14:57:43 GMT

What is it you want regex to do?

Re: Regex with multiple fields duplicated. Yet data can be different?

dmacgillivray — Tue, 13 May 2014 14:59:35 GMT

To be more clear, I just need to write a Regex on this entre output but I cannot get past the duplicated fields as I am just trying to extract this data with some success. All the REGEX training on the planet may not be enough.

This Data output is ClearTrust. The characters have been changed to conceal identities. The length of data has remained the same however.

Re: Regex with multiple fields duplicated. Yet data can be different?

richgalloway — Tue, 13 May 2014 15:33:33 GMT

What fields do you want to extract? What should happen with duplicated fields, extract first, extract last, or extract all? Are you sure you need a regex at all? Perhaps adding MV_ADD=true to your props.conf is enough.

Re: Regex with multiple fields duplicated. Yet data can be different?

dmaislin_splunk — Tue, 13 May 2014 15:44:46 GMT

* | rex  max_match=0 "passwordexpiry_date\:\[(?<Password_Expiry_Date>.+?)\]"

Re: Regex with multiple fields duplicated. Yet data can be different?

dmaislin_splunk — Mon, 28 Sep 2020 16:35:45 GMT

That will give you a field called Password_Expiry_Date with the multiple values per event like you needed. http://docs.splunk.com/Documentation/Splunk/6.0.3/SearchReference/Rex

Re: Regex with multiple fields duplicated. Yet data can be different?

dmacgillivray — Tue, 13 May 2014 15:46:03 GMT

Hi Rich,
Thanks for that information MV_ADD=True so would that allow values to pass through differently?
Unfortunately what I have been given for data has been wrapped by another program. My options are limited. Outside of asking these guys to rewrite their output.

Re: Regex with multiple fields duplicated. Yet data can be different?

dmacgillivray — Tue, 13 May 2014 15:48:15 GMT

Thanks dmaislin this is very helpfull. I will verify it against current output. Looks very promising.

Re: Regex with multiple fields duplicated. Yet data can be different?

dmacgillivray — Tue, 13 May 2014 15:48:39 GMT

Re: Regex with multiple fields duplicated. Yet data can be different?

dmaislin_splunk — Mon, 28 Sep 2020 16:35:50 GMT

| rex max_match=0 "passwordexpiry_date:[(?.+?)]"

That will give you a field called Password_Expiry_Date with the multiple values per event like you needed. http://docs.splunk.com/Documentation/Splunk/6.0.3/SearchReference/Rex

Re: Regex with multiple fields duplicated. Yet data can be different?

richgalloway — Tue, 13 May 2014 15:59:52 GMT

MV_ADD = true will create a multivalue field rather than ignore duplicate field names.

Re: Regex with multiple fields duplicated. Yet data can be different?

dmacgillivray — Tue, 13 May 2014 16:09:20 GMT

Thanks, this is a big help. I will work this into my query. What has been happening is fields that are duplicated just come into the _raw data flow in Splunk. So many good answers. I appreciate every one.