o365:management:activity field extractions

[adalbor]

Has anyone had any success writing field extractions for O365 based events collected via the API?

The messages that are generated are HUGE and have multiple fields that contain multiple values.

I have tried to use eval and mvindex to see if its possible to extract those values but it doesn't appear to be working and I am wondering if its because of the JSON format.

Writing a regex for one of these events would have me ending up with something a page long lol.

Thanks,
Andrew

[to4kawa]

|makeresults
| eval _raw="[{\"CreationTime\":\"2015-06-29T20:03:19\",\"Id\":\"80c76bd2-9d81-4c57-a97a-accfc3443dca\",\"Operation\":\"PasswordLogonInitialAuthUsingPassword\",\"OrganizationId\":\"41463f53-8812-40f4-890f-865bf6e35190\",\"RecordType\":9,\"ResultStatus\":\"failed\",\"UserKey\":\"1153977025279851686@contoso.onmicrosoft.com\",\"UserType\":0,\"Workload\":\"AzureActiveDirectory\",\"ClientIP\":\"134.170.188.221\",\"ObjectId\":\"admin@contoso.onmicrosoft.com\",\"UserId\":\"admin@contoso.onmicrosoft.com\",\"AzureActiveDirectoryEventType\":0,\"ExtendedProperties\":[{\"Name\":\"LoginError\",\"Value\":\"-2147217390;PP_E_BAD_PASSWORD;The entered and stored passwords do not match.\"}],\"Client\":\"Exchange\",\"LoginStatus\":-2147217390,\"UserDomain\":\"contoso.onmicrosoft.com\"},{\"CreationTime\":\"2015-06-29T20:03:34\",\"Id\":\"4e655d3f-35fa-42e0-b050-264b2d255c7a\",\"Operation\":\"PasswordLogonInitialAuthUsingPassword\",\"OrganizationId\":\"41463f53-8812-40f4-890f-865bf6e35190\",\"RecordType\":9,\"ResultStatus\":\"success\",\"UserKey\":\"1153977025279851686@contoso.onmicrosoft.com\",\"UserType\":0,\"Workload\":\"AzureActiveDirectory\",\"ClientIP\":\"134.170.188.221\",\"ObjectId\":\"admin@contoso.onmicrosoft.com\",\"UserId\":\"admin@contoso.onmicrosoft.com\",\"AzureActiveDirectoryEventType\":0,\"Client\":\"Exchange\",\"LoginStatus\":0,\"UserDomain\":\"contoso.onmicrosoft.com\"},{\"CreationTime\":\"2015-06-29T20:04:55\",\"Id\":\"b567caf0-088e-4c1c-a4ea-633a1e3d66c8\",\"Operation\":\"Add User.\",\"OrganizationId\":\"41463f53-8812-40f4-890f-865bf6e35190\",\"RecordType\":8,\"ResultStatus\":\"success\",\"UserKey\":\"1003BFFD8EC47CA6@contoso.onmicrosoft.com\",\"UserType\":0,\"Workload\":\"AzureActiveDirectory\",\"ObjectId\":\"user001@contoso.onmicrosoft.com\",\"UserId\":\"admin@contoso.onmicrosoft.com\",\"AzureActiveDirectoryEventType\":0,\"Actor\":[{\"ID\":\"1cef1fdb-ff52-48c4-8e4e-dfb5ea83d357\",\"Type\":2},{\"ID\":\"admin@contoso.onmicrosoft.com\",\"Type\":5},{\"ID\":\"1003BFFD8EC47CA6\",\"Type\":3}],\"ActorContextId\":\"41463f53-8812-40f4-890f-865bf6e35190\",\"InterSystemsId\":\"c2ced078-ad57-4079-a743-5c37f5284790\",\"IntraSystemId\":\"d1497f7e-15b4-49aa-83ad-11a17ca4a2f4\",\"Target\":[{\"ID\":\"user001@contoso.onmicrosoft.com\",\"Type\":5},{\"ID\":\"10037FFE91510806\",\"Type\":3}],\"TargetContextId\":\"41463f53-8812-40f4-890f-865bf6e35190\"}]"
| spath {} output=root
| stats count by root
| spath input=root Actor{} output=Actor
| mvexpand Actor
| spath input=Actor
| spath input=root Target{} output=Target
| mvexpand Target
| rename Type as Actor_Type
| spath input=Target
| spath input=root
| fields - Actor* root Target* count

https://docs.microsoft.com/en-Us/office/office-365-management-api/office-365-management-activity-api...
That's a lot of work.

If response is Array (not Object) like this, please make props.conf and LINE_BREAKER = (\[|,){\"CreationTime|(\]$)

[adalbor]

Thanks for the assistance. I tested that line_breaker config in our test environment but doesn't appear to be doing anything. It definitely is a lot of work to try and parse these logs. One example log I am looking at is over 32k characters!

[to4kawa]

Do you use SHOULD_LINEMERGE = true ?

[to4kawa]

can you provide sample logs link?