JSON file will not break correctly OR create field extractions

jason_hotchkiss — Thu, 13 May 2021 23:29:35 GMT

inputsHello -

I have the following log that will not line break using the traditional ([\r\n)+). Each event splits between: "Properties": {

Here is what I have tried in my Props.conf:

[ mysourcetype ]
BREAK_ONLY_BEFORE=\"Properties\"\: \{
LINE_BREAKER=^{
CHARSET=UTF-8
DATETIME_CONFIG=CURRENT
MAX_EVENTS=40000
SHOULD_LINEMERGE=true
disabled=false
pulldown_type=true

{ "computers": [ { "Properties": { "haslaps": false, "highvalue": false, "name": "DATA", "domain": "DATA", "objectid": "DATA", "distinguishedname": "DATA", "description": null, "enabled": true, "unconstraineddelegation": false, "serviceprincipalnames": [ "DATA", "DATA", "DATA", "DATA", "DATA", "DATA", "DATA", "DATA" ], "lastlogontimestamp": 1501470433, "pwdlastset": 1500622271, "operatingsystem": "DATA" }, "AllowedToDelegate": [], "AllowedToAct": [], "PrimaryGroupSid": "DATA", "Sessions": [], "LocalAdmins": [], "RemoteDesktopUsers": [], "DcomUsers": [], "PSRemoteUsers": [], "ObjectIdentifier": "DATA", "Aces": [ { "PrincipalSID": "DATA", "PrincipalType": "DATA", "RightName": "DATA", "AceType": "", "IsInherited": DATA }, { "PrincipalSID": "DATA", "PrincipalType": "DATA", "RightName": "DATA", "AceType": "", "IsInherited": false }, { "PrincipalSID": "DATA", "PrincipalType": "DATA", "RightName": "DATA", "AceType": "", "IsInherited": false }, { "PrincipalSID": "DATA", "PrincipalType": "DATA", "RightName": "DATA", "AceType": "", "IsInherited": true }, { "PrincipalSID": "DATA", "PrincipalType": "DATA", "RightName": "DATA", "AceType": "", "IsInherited": true }, { "PrincipalSID": "DATA", "PrincipalType": "DATA", "RightName": "DATA", "AceType": "", "IsInherited": true }, { "PrincipalSID": "DATA", "PrincipalType": "DATA", "RightName": "DATA", "AceType": "", "IsInherited": true }, { "PrincipalSID": "DATA", "PrincipalType": "Unknown", "RightName": "DATA", "AceType": "", "IsInherited": true }, { "PrincipalSID": "DATA", "PrincipalType": "Group", "RightName": "GenericAll", "AceType": "", "IsInherited": true }, { "PrincipalSID": "DATA", "PrincipalType": "Unknown", "RightName": "GenericAll", "AceType": "", "IsInherited": true }, { "PrincipalSID": "DATA", "PrincipalType": "Group", "RightName": "WriteDacl", "AceType": "", "IsInherited": true }, { "PrincipalSID": "DATA", "PrincipalType": "Group", "RightName": "WriteOwner", "AceType": "", "IsInherited": true }, { "PrincipalSID": "DATA", "PrincipalType": "Group", "RightName": "GenericWrite", "AceType": "", "IsInherited": true } ] }, { "Properties": { "haslaps": false, "highvalue": false, "name": "DATA", "domain": "DATA", "objectid": "DATA", "distinguishedname": "DATA", "description": null, "enabled": true, "unconstraineddelegation": false, "serviceprincipalnames": [ "DATA", "DATA", "DATA", "DATA", "DATA", "DATA", "DATA", "DATA", "DATA", "DATA" ], "lastlogontimestamp": 1506599859, "pwdlastset": 1505682659, "operatingsystem": "DATA" }, "AllowedToDelegate": [], "AllowedToAct": [], "PrimaryGroupSid": "DATA", "Sessions": [], "LocalAdmins": [], "RemoteDesktopUsers": [], "DcomUsers": [], "PSRemoteUsers": [], "ObjectIdentifier": "DATA", "Aces": [ { "PrincipalSID": "DATA", "PrincipalType": "Group", "RightName": "Owner", "AceType": "", "IsInherited": false }, { "PrincipalSID": "DATA", "PrincipalType": "Group", "RightName": "GenericAll", "AceType": "", "IsInherited": false }, { "PrincipalSID": "DATA", "PrincipalType": "Group", "RightName": "GenericAll", "AceType": "", "IsInherited": false }, { "PrincipalSID": "DATA", "PrincipalType": "User", "RightName": "GenericAll", "AceType": "", "IsInherited": true }, { "PrincipalSID": "DATA", "PrincipalType": "Group", "RightName": "GenericAll", "AceType": "", "IsInherited": true }, { "PrincipalSID": "DATA", "PrincipalType": "Unknown", "RightName": "GenericAll", "AceType": "", "IsInherited": true }, { "PrincipalSID": "DATA", "PrincipalType": "Group", "RightName": "WriteDacl", "AceType": "", "IsInherited": true }, { "PrincipalSID": "DATA", "PrincipalType": "Group", "RightName": "WriteOwner", "AceType": "", "IsInherited": true }, { "PrincipalSID": "DATA", "PrincipalType": "Group", "RightName": "GenericWrite", "AceType": "", "IsInherited": true } ] <...truncated...>

Any suggestions on how I can get this to break properly & extract the field value pairs? Thank you!

Re: JSON file will not break correctly OR create field extractions

kamlesh_vaghela — Fri, 14 May 2021 05:30:20 GMT

@jason_hotchkiss

Can you please try this?

[YOUR_SOURCETYPE] SHOULD_LINEMERGE=false LINE_BREAKER=]}(\,\s){\"Properties NO_BINARY_CHECK=true SEDCMD-a=s/{\"computers\": \[//g SEDCMD-b=s/\]}\]}$/]}/g TRUNCATE=0

Thanks
Kamlesh Vaghela

Re: JSON file will not break correctly OR create field extractions

jason_hotchkiss — Fri, 14 May 2021 14:30:25 GMT

@kamlesh_vaghela

Hello Kamlesh - that did not work, and I believe that was my fault, as the log format came in wrong.

Here is the regex: https://regex101.com/r/KkpSIM/1

Re: JSON file will not break correctly OR create field extractions

kamlesh_vaghela — Fri, 14 May 2021 17:36:05 GMT

@jason_hotchkiss

Ca you please try this?

[ <SOURCETYPE NAME> ] SHOULD_LINEMERGE=false LINE_BREAKER=\]\n\s{4}}(,\s{5}){\n\s{6}"Properties": NO_BINARY_CHECK=true CHARSET=UTF-8 disabled=false SEDCMD-a=s/{\n\s{2}"computers":\s\[\n\s{4}//g SEDCMD-b=s/\n\s*//g SEDCMD-c=s/\]}\]}$/]}/g

Your provided json was not valid. So I have added closing brackets at the end.

Check the sample I have used.

https://regex101.com/r/NFxrJp/1

Thanks
KV ▄︻̷̿┻̿═━一

If this reply helps you, an upvote would be appreciated.

Re: JSON file will not break correctly OR create field extractions

jason_hotchkiss — Fri, 14 May 2021 20:33:05 GMT

This works! Thank you!

topic Re: JSON file will not break correctly OR create field extractions in Getting Data In

JSON file will not break correctly OR create field extractions

Re: JSON file will not break correctly OR create field extractions

Re: JSON file will not break correctly OR create field extractions

Re: JSON file will not break correctly OR create field extractions

Re: JSON file will not break correctly OR create field extractions