Splunk Search

Regex with multiple fields duplicated. Yet data can be different?

dmacgillivray
Communicator

I have an issue with data titles that would appear to be repeated, yet in the case below, The passwordexpiry_date: field can be repeated with different values.

See below account fields are replicated and I am trying to determine if a REGEX can even be written on this. Especially when field names are duplicated.

sequence_number=18112355,
remote_client=Eserver,
2014-05-08 18:02:47:84 GMT,
conn=72276,
op=239,
eventID=0a46b98715e113924708169294656,
messageID=701,
ip=127.0.0.1,
uname=gaspm05,
urole=Bobs And Admin,
msg=Modify user,
msgtype=MODIFY,
result=0,
etime=34ms,
upd_user_owner=Person User Admin Group,
associated_parent_member_groups=[ ],
upd_user=DATA_BOB_Truck3bobsurunc,
new_user=[userid:
[DATA_BOB_Truck3bobsurunc]
firstname:[Truck3bobsurunc]
lastname:[LN]
email:[]
certdn:[]
properties:[[name=ContactID isset=false], [name=ctscPasswordResetAttempts isset=false], [name=ctscSecretQuestionAnswer isset=false]]
account_start_date:[Mar 12, 2014 6:08:47 PM]
account_expiry_date:[Jan 1, 4712 6:08:47 PM]
islockedout:[false]
isactive:[true]
passwordexpiry_date:[Aug 6, 2014 6:02:47 PM]
admin_group:[Person User Admin Group]
ispublic:[true]],
original_user=[userid:[DATA_BOB_Truck3bobsurunc]
firstname:[Truck3bobsurunc]
lastname:[LN]
email:[]
certdn:[]
properties:[[name=ContactID isset=false], [name=ctscPasswordResetAttempts isset=false], [name=ctscSecretQuestionAnswer isset=false]]
account_start_date:[Mar 12, 2014 6:08:47 PM]
account_expiry_date:[Jan 1, 4712 6:08:47 PM]
islockedout:[false]
isactive:[true]
passwordexpiry_date:[Jul 30, 2014 6:26:37 PM]
admin_group:[Person User Admin Group]
ispublic:[true]],
user_owner=Person User Admin Group,
user=DATA_BOB_Truck3bobsurunc,
userUID=CCCCCQEBBBBDU1FMJJJJBlRlc3R2MgAAAAEQDDDDDDDkjBE=

Tags (3)
0 Karma
1 Solution

dmaislin_splunk
Splunk Employee
Splunk Employee
* | rex  max_match=0 "passwordexpiry_date\:\[(?<Password_Expiry_Date>.+?)\]"

View solution in original post

dmacgillivray
Communicator

Thanks, this is a big help. I will work this into my query. What has been happening is fields that are duplicated just come into the _raw data flow in Splunk. So many good answers. I appreciate every one.

0 Karma

richgalloway
SplunkTrust
SplunkTrust

MV_ADD = true will create a multivalue field rather than ignore duplicate field names.

---
If this reply helps you, Karma would be appreciated.
0 Karma

dmaislin_splunk
Splunk Employee
Splunk Employee
  • | rex max_match=0 "passwordexpiry_date:[(?.+?)]"

That will give you a field called Password_Expiry_Date with the multiple values per event like you needed. http://docs.splunk.com/Documentation/Splunk/6.0.3/SearchReference/Rex

0 Karma

dmacgillivray
Communicator

Hi Rich,
Thanks for that information MV_ADD=True so would that allow values to pass through differently?
Unfortunately what I have been given for data has been wrapped by another program. My options are limited. Outside of asking these guys to rewrite their output.

0 Karma

dmacgillivray
Communicator

Thanks dmaislin this is very helpfull. I will verify it against current output. Looks very promising.

0 Karma

dmacgillivray
Communicator

Hi Rich,
Thanks for that information MV_ADD=True so would that allow values to pass through differently?
Unfortunately what I have been given for data has been wrapped by another program. My options are limited. Outside of asking these guys to rewrite their output.

0 Karma

dmaislin_splunk
Splunk Employee
Splunk Employee
* | rex  max_match=0 "passwordexpiry_date\:\[(?<Password_Expiry_Date>.+?)\]"

dmaislin_splunk
Splunk Employee
Splunk Employee

That will give you a field called Password_Expiry_Date with the multiple values per event like you needed. http://docs.splunk.com/Documentation/Splunk/6.0.3/SearchReference/Rex

0 Karma

richgalloway
SplunkTrust
SplunkTrust

What fields do you want to extract? What should happen with duplicated fields, extract first, extract last, or extract all? Are you sure you need a regex at all? Perhaps adding MV_ADD=true to your props.conf is enough.

---
If this reply helps you, Karma would be appreciated.
0 Karma

dmacgillivray
Communicator

To be more clear, I just need to write a Regex on this entre output but I cannot get past the duplicated fields as I am just trying to extract this data with some success. All the REGEX training on the planet may not be enough.

This Data output is ClearTrust. The characters have been changed to conceal identities. The length of data has remained the same however.

0 Karma

richgalloway
SplunkTrust
SplunkTrust

What is it you want regex to do?

---
If this reply helps you, Karma would be appreciated.
0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...