Splunk Search
×

Are you a member of the Splunk Community?

Sign in or Register with your Splunk account to get your questions answered, access valuable resources and connect with experts!
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Show  only  | Search instead for 
Did you mean: 
Ask a Question

Combining events over time

jparso09
New Member
‎02-06-2025 04:43 PM

I am not sure where to even start on this one. 

 

I have 2 log file types I need to extract data to get final accounts. I need to combine by objectClasses so that when on a given day "ial to enforce" in log Type 2 is sets the count for number of Type 1 events. I need to run this over a year. Thank you in advance!!!! 


-----Type 1

2025-01-01 00:00:00,125 trackingid="tid:13256464"message='{"UserAccessSubmission":{"uuid":"abc123","mail":"sean@southpark.net","trackingId":"tid:13256464","objectClass":"cartmanUser","csp":"Butters"}}'
2025-01-01 00:01:00,125 trackingid="tid:13256464"message='{"UserAccessSubmission":{"uuid":"abc123","mail":"sean@southpark.net","trackingId":"tid:13256464","objectClass":"cartmanUser","csp":"Butters"}}'

2025-01-02 00:01:00,125 trackingid="tid:13256464"message='{"UserAccessSubmission":{"uuid":"abc123","mail":"sean@southpark.net","trackingId":"tid:13256464","objectClass":"cartmanUser","csp":"Butters"}}'

2025-01-02 00:01:00,125 trackingid="tid:13256464"message='{"UserAccessSubmission":{"uuid":"abc123","mail":"sean@southpark.net","trackingId":"tid:13256464","objectClass":"StanUser","csp":"Butters"}}'

2025-01-02 00:01:00,125 trackingid="tid:13256464"message='{"UserAccessSubmission":{"uuid":"abc123","mail":"sean@southpark.net","trackingId":"tid:13256464","objectClass":"StanUser","csp":"Butters"}}'

 

------- Type 2

{ [-]
@message: { [-]
attributeContract: { [-]
extendedAttributes: [ [-]
]
maskOgnlValues: false
uniqueUserKeyAttribute: uuid
}
attributeMapping: { [-]
attributeContractFulfillment: { [-]
uuid: { [-]
source: { [-]
type: ADAPTER
}
value: uuid
}
}
attributeSources: [ [-]
]
issuanceCriteria: { [-]
conditionalCriteria: [ [-]
]
}
}
configuration: { [-]
fields: [ [-]
{ [-]
name: Application ObjectClass
value: cartmanUser
}
{ [-]
name: Application Entitlement Attribute
value: cartmanRole
}

{ [-]
name: IAL to Enforce
value: 2
}
}
id: Cartman
name: Cartman
}
@timestamp: 2025-01-01T00:00:01.833685
}

 

{ [-]
@message: { [-]
attributeContract: { [-]
extendedAttributes: [ [-]
]
maskOgnlValues: false
uniqueUserKeyAttribute: uuid
}
attributeMapping: { [-]
attributeContractFulfillment: { [-]
uuid: { [-]
source: { [-]
type: ADAPTER
}
value: uuid
}
}
attributeSources: [ [-]
]
issuanceCriteria: { [-]
conditionalCriteria: [ [-]
]
}
}
configuration: { [-]
fields: [ [-]
{ [-]
name: Application ObjectClass
value: cartmanUser
}
{ [-]
name: Application Entitlement Attribute
value: cartmanRole
}

{ [-]
name: IAL to Enforce
value: 1
}
}
id: Cartman
name: Cartman
}
@timestamp: 2025-01-02T00:00:01.833685
}

 

 

The Goal would be to get something like this

Table 1

 Ial to enforce is 2
CartmanUser2

 

 

Table 2

 Ial to enforce is 1
CartmanUser1

 

Labels (3)
Labels
0 Karma
Reply

yuanliu
SplunkTrust
SplunkTrust
‎02-07-2025 12:34 AM

First, when posting type 2 which is in JSON, please use raw text.  Splunk's "syntax highlights" view is non-compliant and very difficult to process. (See the crazy rex in my emulation below; you also introduced additional syntax errors when attempting to simplify or anonymize.)  Also in type 2, you should preserve the uuid's value as that's the only key that distinguishes between the two.  For everyone's benefit, I'm posting reconstructed raw events from type 2:

 

{ 
"@message": { 
"attributeContract": { 
"extendedAttributes": [ 
],
"maskOgnlValues": false,
"uniqueUserKeyAttribute": "uuid"
},
"attributeMapping": { 
"attributeContractFulfillment": { 
"uuid": { 
"source": { 
"type": "ADAPTER"
},
"value": "9c5b94b1-35ad-49bb-b118-8e8fc24abf80"
}
},
"attributeSources": [ 
],
"issuanceCriteria": { 
"conditionalCriteria": [ 
]
}
},
"configuration": { 
"fields": [ 
{ 
"name": "Application ObjectClass",
"value": "cartmanUser"
},
{ 
"name": "Application Entitlement Attribute",
"value": "cartmanRole"
},
{ 
"name": "IAL to Enforce",
"value": 2
}
],
"id": "Cartman",
"name": "Cartman"
}
},
"@timestamp": "2025-01-01T00:00:01.833685"
}

{ 
"@message": { 
"attributeContract": { 
"extendedAttributes": [ 
],
"maskOgnlValues": false,
"uniqueUserKeyAttribute": "uuid"
},
"attributeMapping": { 
"attributeContractFulfillment": { 
"uuid": { 
"source": { 
"type": "ADAPTER"
},
"value": "550e8400-e29b-41d4-a716-446655440000"
}
},
"attributeSources": [ 
],
"issuanceCriteria": { 
"conditionalCriteria": [ 
]
}
},
"configuration": { 
"fields": [ 
{ 
"name": "Application ObjectClass",
"value": "cartmanUser"
},
{ 
"name": "Application Entitlement Attribute",
"value": "cartmanRole"
},
{ 
"name": "IAL to Enforce",
"value": 1
}
],
"id": "Cartman",
"name": "Cartman"
}
},
"@timestamp": "2025-01-02T00:00:01.833685"
}

 

Like @bowesmana, I fail to see see the relevance of type 1.  Type 2 is all you need to produce the results you want.  I also don't see why you want to print two tables rather than printing one table with two rows (differentiated by UUID).  So, this is what I'm going to show.

Actual code is pretty simple.  My main time was sunken in reconstruct valid JSON data from your pasted text.

 

| fields @message.attributeMapping.attributeContractFulfillment.uuid.value
``` ^^^ this line is just to declutter output ```
| spath path=@message.configuration.fields{}
| eval restructured_fields = json_object()
| foreach @message.configuration.fields{} mode=multivalue
    [eval restructured_fields = json_set(restructured_fields,
     json_extract(<<ITEM>>, "name"), json_extract(<<ITEM>>, "value"))]
| spath input=restructured_fields

 

(This foreach syntax above requires Splunk 9.0.)  Output from the two reconstructed events is as follows:


@message.attributeMapping.attributeContractFulfillment.uuid.value		Application Entitlement AttributeApplication ObjectClassIAL to Enforce
9c5b94b1-35ad-49bb-b118-8e8fc24abf80cartmanRolecartmanUser2
550e8400-e29b-41d4-a716-446655440000cartmanRolecartmanUser1

Does this satisfy your requirements?

It is useful to print out the two intermediate JSON objects used in this search so you can clearly see dataflow:

@message.configuration.fields{}
restructured_fields
{ "name": "Application ObjectClass", "value": "cartmanUser" }
{ "name": "Application Entitlement Attribute", "value": "cartmanRole" }
{ "name": "IAL to Enforce", "value": 2 }
{"Application ObjectClass":"cartmanUser","Application Entitlement Attribute":"cartmanRole","IAL to Enforce":2}
{ "name": "Application ObjectClass", "value": "cartmanUser" }
{ "name": "Application Entitlement Attribute", "value": "cartmanRole" }
{ "name": "IAL to Enforce", "value": 1 }
{"Application ObjectClass":"cartmanUser","Application Entitlement Attribute":"cartmanRole","IAL to Enforce":1}

@message.configuration.fields{}, of source, is extracted directly from raw data.

Here is an emulation for you to play with and compare with real data type 2:

 

| makeresults
| fields - _time
| eval sourcetype = "type2", data = mvappend("{ [-]
@message: { [-]
attributeContract: { [-]
extendedAttributes: [ [-]
]
maskOgnlValues: false
uniqueUserKeyAttribute: uuid
}
attributeMapping: { [-]
attributeContractFulfillment: { [-]
uuid: { [-]
source: { [-]
type: ADAPTER
}
value: 9c5b94b1-35ad-49bb-b118-8e8fc24abf80
}
}
attributeSources: [ [-]
]
issuanceCriteria: { [-]
conditionalCriteria: [ [-]
]
}
}
configuration: { [-]
fields: [ [-]
{ [-]
name: Application ObjectClass
value: cartmanUser
}
{ [-]
name: Application Entitlement Attribute
value: cartmanRole
}
{ [-]
name: IAL to Enforce
value: 2
}
]
id: Cartman
name: Cartman
}
}
@timestamp: 2025-01-01T00:00:01.833685
}",
"{ [-]
@message: { [-]
attributeContract: { [-]
extendedAttributes: [ [-]
]
maskOgnlValues: false
uniqueUserKeyAttribute: uuid
}
attributeMapping: { [-]
attributeContractFulfillment: { [-]
uuid: { [-]
source: { [-]
type: ADAPTER
}
value: 550e8400-e29b-41d4-a716-446655440000
}
}
attributeSources: [ [-]
]
issuanceCriteria: { [-]
conditionalCriteria: [ [-]
]
}
}
configuration: { [-]
fields: [ [-]
{ [-]
name: Application ObjectClass
value: cartmanUser
}
{ [-]
name: Application Entitlement Attribute
value: cartmanRole
}
{ [-]
name: IAL to Enforce
value: 1
}
]
id: Cartman
name: Cartman
}
}
@timestamp: 2025-01-02T00:00:01.833685
}")
| rex field=data mode=sed "s/\[-]//g s/\n+([\w@])/\n\"\1/g s/([^\"]): (true|false|\d+\n)/\1\": \2/g
  s/([^\"]):(\W+\n)/\1\":\2/g s/([^\"]): (.+)/\1\": \"\2\"/g s/([\w\"}\]])\n([\"{\[])/\1,\n\2/g"
| mvexpand data
| rename data AS _raw
| spath
``` data type 2 emulation above ```

(Can you see how crazy that rex command is?)

 

For completeness, this is how you extract data from type 1 in case it is of use to you:

 

| eval message = replace(message, "'", "")
| spath input=message

 

message field should have been present at search type.  The result from your sample data is

UserAccessSubmission.cspUserAccessSubmission.mailUserAccessSubmission.objectClassUserAccessSubmission.trackingIdUserAccessSubmission.uuidsourcetypetrackingid
Butterssean@southpark.netcartmanUsertid:13256464abc123type1tid:13256464
Butterssean@southpark.netcartmanUsertid:13256464abc123type1tid:13256464
Butterssean@southpark.netcartmanUsertid:13256464abc123type1tid:13256464
Butterssean@southpark.netStanUsertid:13256464abc123type1tid:13256464
Butterssean@southpark.netStanUsertid:13256464abc123type1tid:13256464

This is emulation of data type 1 used to extract the above.

 

| makeresults
| fields - _time
| eval sourcetype = "type1", data = split("2025-01-01 00:00:00,125 trackingid=\"tid:13256464\"message='{\"UserAccessSubmission\":{\"uuid\":\"abc123\",\"mail\":\"sean@southpark.net\",\"trackingId\":\"tid:13256464\",\"objectClass\":\"cartmanUser\",\"csp\":\"Butters\"}}'
2025-01-01 00:01:00,125 trackingid=\"tid:13256464\"message='{\"UserAccessSubmission\":{\"uuid\":\"abc123\",\"mail\":\"sean@southpark.net\",\"trackingId\":\"tid:13256464\",\"objectClass\":\"cartmanUser\",\"csp\":\"Butters\"}}'
2025-01-02 00:01:00,125 trackingid=\"tid:13256464\"message='{\"UserAccessSubmission\":{\"uuid\":\"abc123\",\"mail\":\"sean@southpark.net\",\"trackingId\":\"tid:13256464\",\"objectClass\":\"cartmanUser\",\"csp\":\"Butters\"}}'
2025-01-02 00:01:00,125 trackingid=\"tid:13256464\"message='{\"UserAccessSubmission\":{\"uuid\":\"abc123\",\"mail\":\"sean@southpark.net\",\"trackingId\":\"tid:13256464\",\"objectClass\":\"StanUser\",\"csp\":\"Butters\"}}'
2025-01-02 00:01:00,125 trackingid=\"tid:13256464\"message='{\"UserAccessSubmission\":{\"uuid\":\"abc123\",\"mail\":\"sean@southpark.net\",\"trackingId\":\"tid:13256464\",\"objectClass\":\"StanUser\",\"csp\":\"Butters\"}}'", "
")
| mvexpand data
| rename data AS _raw
| extract
``` data type 1 emulation above ```

 

 

Tags (2)
0 Karma
Reply

bowesmana
SplunkTrust
SplunkTrust
‎02-06-2025 07:47 PM

I'm not clear on what the data in type 1 is used for. You have object class in type 2 source as well as type 1 - I assume CartmanUser is meant to be cartmanUser

What are the IAL to enforce values 1 and 2 supposed to correlate to, to give you the 1 and 2 in your tables?

 

0 Karma
Reply
Get Updates on the Splunk Community!

Wrapping Up Cybersecurity Awareness Month

October might be wrapping up, but for Splunk Education, cybersecurity awareness never goes out of season. ...

🌟 From Audit Chaos to Clarity: Welcoming Audit Trail v2

&#x1f5e3; You Spoke, We Listened  Audit Trail v2 wasn’t written in isolation—it was shaped by your voices.  In ...

What's New in Splunk Observability - October 2025

What’s New?    We’re excited to announce the latest enhancements to Splunk Observability Cloud and share ...