Combining events over time

jparso09 — Fri, 07 Feb 2025 00:43:58 GMT

I am not sure where to even start on this one.

I have 2 log file types I need to extract data to get final accounts. I need to combine by objectClasses so that when on a given day "ial to enforce" in log Type 2 is sets the count for number of Type 1 events. I need to run this over a year. Thank you in advance!!!!

-----Type 1

2025-01-01 00:00:00,125 trackingid="tid:13256464"message='{"UserAccessSubmission":{"uuid":"abc123","mail":"sean@southpark.net","trackingId":"tid:13256464","objectClass":"cartmanUser","csp":"Butters"}}'
2025-01-01 00:01:00,125 trackingid="tid:13256464"message='{"UserAccessSubmission":{"uuid":"abc123","mail":"sean@southpark.net","trackingId":"tid:13256464","objectClass":"cartmanUser","csp":"Butters"}}'

2025-01-02 00:01:00,125 trackingid="tid:13256464"message='{"UserAccessSubmission":{"uuid":"abc123","mail":"sean@southpark.net","trackingId":"tid:13256464","objectClass":"cartmanUser","csp":"Butters"}}'

2025-01-02 00:01:00,125 trackingid="tid:13256464"message='{"UserAccessSubmission":{"uuid":"abc123","mail":"sean@southpark.net","trackingId":"tid:13256464","objectClass":"StanUser","csp":"Butters"}}'

------- Type 2

{ [-]
@message: { [-]
attributeContract: { [-]
extendedAttributes: [ [-]
]
maskOgnlValues: false
uniqueUserKeyAttribute: uuid
}
attributeMapping: { [-]
attributeContractFulfillment: { [-]
uuid: { [-]
source: { [-]
type: ADAPTER
}
value: uuid
}
}
attributeSources: [ [-]
]
issuanceCriteria: { [-]
conditionalCriteria: [ [-]
]
}
}
configuration: { [-]
fields: [ [-]
{ [-]
name: Application ObjectClass
value: cartmanUser
}
{ [-]
name: Application Entitlement Attribute
value: cartmanRole
}

{ [-]
name: IAL to Enforce
value: 2
}
}
id: Cartman
name: Cartman
}
@timestamp: 2025-01-01T00:00:01.833685
}

{ [-]
name: IAL to Enforce
value: 1
}
}
id: Cartman
name: Cartman
}
@timestamp: 2025-01-02T00:00:01.833685
}

The Goal would be to get something like this

Table 1

	Ial to enforce is 2
CartmanUser	2

Table 2

	Ial to enforce is 1
CartmanUser	1

Re: Combining events over time

bowesmana — Fri, 07 Feb 2025 03:47:31 GMT

I'm not clear on what the data in type 1 is used for. You have object class in type 2 source as well as type 1 - I assume CartmanUser is meant to be cartmanUser

What are the IAL to enforce values 1 and 2 supposed to correlate to, to give you the 1 and 2 in your tables?

Re: Combining events over time

yuanliu — Fri, 07 Feb 2025 08:34:20 GMT

First, when posting type 2 which is in JSON, please use raw text. Splunk's "syntax highlights" view is non-compliant and very difficult to process. (See the crazy rex in my emulation below; you also introduced additional syntax errors when attempting to simplify or anonymize.) Also in type 2, you should preserve the uuid's value as that's the only key that distinguishes between the two. For everyone's benefit, I'm posting reconstructed raw events from type 2:

{ "@message": { "attributeContract": { "extendedAttributes": [ ], "maskOgnlValues": false, "uniqueUserKeyAttribute": "uuid" }, "attributeMapping": { "attributeContractFulfillment": { "uuid": { "source": { "type": "ADAPTER" }, "value": "9c5b94b1-35ad-49bb-b118-8e8fc24abf80" } }, "attributeSources": [ ], "issuanceCriteria": { "conditionalCriteria": [ ] } }, "configuration": { "fields": [ { "name": "Application ObjectClass", "value": "cartmanUser" }, { "name": "Application Entitlement Attribute", "value": "cartmanRole" }, { "name": "IAL to Enforce", "value": 2 } ], "id": "Cartman", "name": "Cartman" } }, "@timestamp": "2025-01-01T00:00:01.833685" } { "@message": { "attributeContract": { "extendedAttributes": [ ], "maskOgnlValues": false, "uniqueUserKeyAttribute": "uuid" }, "attributeMapping": { "attributeContractFulfillment": { "uuid": { "source": { "type": "ADAPTER" }, "value": "550e8400-e29b-41d4-a716-446655440000" } }, "attributeSources": [ ], "issuanceCriteria": { "conditionalCriteria": [ ] } }, "configuration": { "fields": [ { "name": "Application ObjectClass", "value": "cartmanUser" }, { "name": "Application Entitlement Attribute", "value": "cartmanRole" }, { "name": "IAL to Enforce", "value": 1 } ], "id": "Cartman", "name": "Cartman" } }, "@timestamp": "2025-01-02T00:00:01.833685" }

Like @bowesmana, I fail to see see the relevance of type 1. Type 2 is all you need to produce the results you want. I also don't see why you want to print two tables rather than printing one table with two rows (differentiated by UUID). So, this is what I'm going to show.

Actual code is pretty simple. My main time was sunken in reconstruct valid JSON data from your pasted text.

| fields @message.attributeMapping.attributeContractFulfillment.uuid.value ``` ^^^ this line is just to declutter output ``` | spath path=@message.configuration.fields{} | eval restructured_fields = json_object() | foreach @message.configuration.fields{} mode=multivalue [eval restructured_fields = json_set(restructured_fields, json_extract(<<ITEM>>, "name"), json_extract(<<ITEM>>, "value"))] | spath input=restructured_fields

(This foreach syntax above requires Splunk 9.0.) Output from the two reconstructed events is as follows:

@message.attributeMapping.attributeContractFulfillment.uuid.value	Application Entitlement Attribute	Application ObjectClass	IAL to Enforce
9c5b94b1-35ad-49bb-b118-8e8fc24abf80	cartmanRole	cartmanUser	2
550e8400-e29b-41d4-a716-446655440000	cartmanRole	cartmanUser	1

Does this satisfy your requirements?

It is useful to print out the two intermediate JSON objects used in this search so you can clearly see dataflow:

@message.configuration.fields{}	restructured_fields
{ "name": "Application ObjectClass", "value": "cartmanUser" } { "name": "Application Entitlement Attribute", "value": "cartmanRole" } { "name": "IAL to Enforce", "value": 2 }	{"Application ObjectClass":"cartmanUser","Application Entitlement Attribute":"cartmanRole","IAL to Enforce":2}
{ "name": "Application ObjectClass", "value": "cartmanUser" } { "name": "Application Entitlement Attribute", "value": "cartmanRole" } { "name": "IAL to Enforce", "value": 1 }	{"Application ObjectClass":"cartmanUser","Application Entitlement Attribute":"cartmanRole","IAL to Enforce":1}

@message.configuration.fields{}, of source, is extracted directly from raw data.

Here is an emulation for you to play with and compare with real data type 2:

| makeresults | fields - _time | eval sourcetype = "type2", data = mvappend("{ [-] @message: { [-] attributeContract: { [-] extendedAttributes: [ [-] ] maskOgnlValues: false uniqueUserKeyAttribute: uuid } attributeMapping: { [-] attributeContractFulfillment: { [-] uuid: { [-] source: { [-] type: ADAPTER } value: 9c5b94b1-35ad-49bb-b118-8e8fc24abf80 } } attributeSources: [ [-] ] issuanceCriteria: { [-] conditionalCriteria: [ [-] ] } } configuration: { [-] fields: [ [-] { [-] name: Application ObjectClass value: cartmanUser } { [-] name: Application Entitlement Attribute value: cartmanRole } { [-] name: IAL to Enforce value: 2 } ] id: Cartman name: Cartman } } @timestamp: 2025-01-01T00:00:01.833685 }", "{ [-] @message: { [-] attributeContract: { [-] extendedAttributes: [ [-] ] maskOgnlValues: false uniqueUserKeyAttribute: uuid } attributeMapping: { [-] attributeContractFulfillment: { [-] uuid: { [-] source: { [-] type: ADAPTER } value: 550e8400-e29b-41d4-a716-446655440000 } } attributeSources: [ [-] ] issuanceCriteria: { [-] conditionalCriteria: [ [-] ] } } configuration: { [-] fields: [ [-] { [-] name: Application ObjectClass value: cartmanUser } { [-] name: Application Entitlement Attribute value: cartmanRole } { [-] name: IAL to Enforce value: 1 } ] id: Cartman name: Cartman } } @timestamp: 2025-01-02T00:00:01.833685 }") | rex field=data mode=sed "s/\[-]//g s/\n+([\w@])/\n\"\1/g s/([^\"]): (true|false|\d+\n)/\1\": \2/g s/([^\"]):(\W+\n)/\1\":\2/g s/([^\"]): (.+)/\1\": \"\2\"/g s/([\w\"}\]])\n([\"{\[])/\1,\n\2/g" | mvexpand data | rename data AS _raw | spath ``` data type 2 emulation above ```

(Can you see how crazy that rex command is?)

For completeness, this is how you extract data from type 1 in case it is of use to you:

| eval message = replace(message, "'", "") | spath input=message

message field should have been present at search type. The result from your sample data is

UserAccessSubmission.csp	UserAccessSubmission.mail	UserAccessSubmission.objectClass	UserAccessSubmission.trackingId	UserAccessSubmission.uuid	sourcetype	trackingid
Butters	sean@southpark.net	cartmanUser	tid:13256464	abc123	type1	tid:13256464
Butters	sean@southpark.net	cartmanUser	tid:13256464	abc123	type1	tid:13256464
Butters	sean@southpark.net	cartmanUser	tid:13256464	abc123	type1	tid:13256464
Butters	sean@southpark.net	StanUser	tid:13256464	abc123	type1	tid:13256464
Butters	sean@southpark.net	StanUser	tid:13256464	abc123	type1	tid:13256464

This is emulation of data type 1 used to extract the above.

| makeresults | fields - _time | eval sourcetype = "type1", data = split("2025-01-01 00:00:00,125 trackingid=\"tid:13256464\"message='{\"UserAccessSubmission\":{\"uuid\":\"abc123\",\"mail\":\"sean@southpark.net\",\"trackingId\":\"tid:13256464\",\"objectClass\":\"cartmanUser\",\"csp\":\"Butters\"}}' 2025-01-01 00:01:00,125 trackingid=\"tid:13256464\"message='{\"UserAccessSubmission\":{\"uuid\":\"abc123\",\"mail\":\"sean@southpark.net\",\"trackingId\":\"tid:13256464\",\"objectClass\":\"cartmanUser\",\"csp\":\"Butters\"}}' 2025-01-02 00:01:00,125 trackingid=\"tid:13256464\"message='{\"UserAccessSubmission\":{\"uuid\":\"abc123\",\"mail\":\"sean@southpark.net\",\"trackingId\":\"tid:13256464\",\"objectClass\":\"cartmanUser\",\"csp\":\"Butters\"}}' 2025-01-02 00:01:00,125 trackingid=\"tid:13256464\"message='{\"UserAccessSubmission\":{\"uuid\":\"abc123\",\"mail\":\"sean@southpark.net\",\"trackingId\":\"tid:13256464\",\"objectClass\":\"StanUser\",\"csp\":\"Butters\"}}' 2025-01-02 00:01:00,125 trackingid=\"tid:13256464\"message='{\"UserAccessSubmission\":{\"uuid\":\"abc123\",\"mail\":\"sean@southpark.net\",\"trackingId\":\"tid:13256464\",\"objectClass\":\"StanUser\",\"csp\":\"Butters\"}}'", " ") | mvexpand data | rename data AS _raw | extract ``` data type 1 emulation above ```

topic Re: Combining events over time in Splunk Search

Combining events over time

Re: Combining events over time

Re: Combining events over time