Regex with multiple fields duplicated. Yet data can be different?

[dmacgillivray]

I have an issue with data titles that would appear to be repeated, yet in the case below, The passwordexpiry_date: field can be repeated with different values.

See below account fields are replicated and I am trying to determine if a REGEX can even be written on this. Especially when field names are duplicated.

sequence_number=18112355,
remote_client=Eserver,
2014-05-08 18:02:47:84 GMT,
conn=72276,
op=239,
eventID=0a46b98715e113924708169294656,
messageID=701,
ip=127.0.0.1,
uname=gaspm05,
urole=Bobs And Admin,
msg=Modify user,
msgtype=MODIFY,
result=0,
etime=34ms,
upd_user_owner=Person User Admin Group,
associated_parent_member_groups=[ ],
upd_user=DATA_BOB_Truck3bobsurunc,
new_user=[userid:
[DATA_BOB_Truck3bobsurunc]
firstname:[Truck3bobsurunc]
lastname:[LN]
email:[]
certdn:[]
properties:[[name=ContactID isset=false], [name=ctscPasswordResetAttempts isset=false], [name=ctscSecretQuestionAnswer isset=false]]
account_start_date:[Mar 12, 2014 6:08:47 PM]
account_expiry_date:[Jan 1, 4712 6:08:47 PM]
islockedout:[false]
isactive:[true]
passwordexpiry_date:[Aug 6, 2014 6:02:47 PM]
admin_group:[Person User Admin Group]
ispublic:[true]],
original_user=[userid:[DATA_BOB_Truck3bobsurunc]
firstname:[Truck3bobsurunc]
lastname:[LN]
email:[]
certdn:[]
properties:[[name=ContactID isset=false], [name=ctscPasswordResetAttempts isset=false], [name=ctscSecretQuestionAnswer isset=false]]
account_start_date:[Mar 12, 2014 6:08:47 PM]
account_expiry_date:[Jan 1, 4712 6:08:47 PM]
islockedout:[false]
isactive:[true]
passwordexpiry_date:[Jul 30, 2014 6:26:37 PM]
admin_group:[Person User Admin Group]
ispublic:[true]],
user_owner=Person User Admin Group,
user=DATA_BOB_Truck3bobsurunc,
userUID=CCCCCQEBBBBDU1FMJJJJBlRlc3R2MgAAAAEQDDDDDDDkjBE=

[dmacgillivray]

Thanks, this is a big help. I will work this into my query. What has been happening is fields that are duplicated just come into the _raw data flow in Splunk. So many good answers. I appreciate every one.

[richgalloway]

MV_ADD = true will create a multivalue field rather than ignore duplicate field names.

[dmaislin_splunk]

• | rex max_match=0 "passwordexpiry_date:[(?.+?)]"

That will give you a field called Password_Expiry_Date with the multiple values per event like you needed. http://docs.splunk.com/Documentation/Splunk/6.0.3/SearchReference/Rex

[dmacgillivray]

Hi Rich,
Thanks for that information MV_ADD=True so would that allow values to pass through differently?
Unfortunately what I have been given for data has been wrapped by another program. My options are limited. Outside of asking these guys to rewrite their output.

[dmacgillivray]

Thanks dmaislin this is very helpfull. I will verify it against current output. Looks very promising.

[dmacgillivray]

Hi Rich,
Thanks for that information MV_ADD=True so would that allow values to pass through differently?
Unfortunately what I have been given for data has been wrapped by another program. My options are limited. Outside of asking these guys to rewrite their output.

[dmaislin_splunk · Accepted solution]

* | rex  max_match=0 "passwordexpiry_date\:\[(?<Password_Expiry_Date>.+?)\]"

[dmaislin_splunk]

That will give you a field called Password_Expiry_Date with the multiple values per event like you needed. http://docs.splunk.com/Documentation/Splunk/6.0.3/SearchReference/Rex

[richgalloway]

What fields do you want to extract? What should happen with duplicated fields, extract first, extract last, or extract all? Are you sure you need a regex at all? Perhaps adding MV_ADD=true to your props.conf is enough.

[dmacgillivray]

To be more clear, I just need to write a Regex on this entre output but I cannot get past the duplicated fields as I am just trying to extract this data with some success. All the REGEX training on the planet may not be enough.

This Data output is ClearTrust. The characters have been changed to conceal identities. The length of data has remained the same however.

[richgalloway]

What is it you want regex to do?