Getting Data In

JSON file will not break correctly OR create field extractions

jason_hotchkiss
Communicator

inputsHello - 

I have the following log that will not line break using the traditional ([\r\n)+).  Each event splits between:  "Properties": {

Here is what I have tried in my Props.conf:

[ mysourcetype ]
BREAK_ONLY_BEFORE=\"Properties\"\: \{
LINE_BREAKER=^{
CHARSET=UTF-8
DATETIME_CONFIG=CURRENT
MAX_EVENTS=40000
SHOULD_LINEMERGE=true
disabled=false
pulldown_type=true


{ "computers": [ { "Properties": { "haslaps": false, "highvalue": false, "name": "DATA", "domain": "DATA", "objectid": "DATA", "distinguishedname": "DATA", "description": null, "enabled": true, "unconstraineddelegation": false, "serviceprincipalnames": [ "DATA", "DATA", "DATA", "DATA", "DATA", "DATA", "DATA", "DATA" ], "lastlogontimestamp": 1501470433, "pwdlastset": 1500622271, "operatingsystem": "DATA" }, "AllowedToDelegate": [], "AllowedToAct": [], "PrimaryGroupSid": "DATA", "Sessions": [], "LocalAdmins": [], "RemoteDesktopUsers": [], "DcomUsers": [], "PSRemoteUsers": [], "ObjectIdentifier": "DATA", "Aces": [ { "PrincipalSID": "DATA", "PrincipalType": "DATA", "RightName": "DATA", "AceType": "", "IsInherited": DATA }, { "PrincipalSID": "DATA", "PrincipalType": "DATA", "RightName": "DATA", "AceType": "", "IsInherited": false }, { "PrincipalSID": "DATA", "PrincipalType": "DATA", "RightName": "DATA", "AceType": "", "IsInherited": false }, { "PrincipalSID": "DATA", "PrincipalType": "DATA", "RightName": "DATA", "AceType": "", "IsInherited": true }, { "PrincipalSID": "DATA", "PrincipalType": "DATA", "RightName": "DATA", "AceType": "", "IsInherited": true }, { "PrincipalSID": "DATA", "PrincipalType": "DATA", "RightName": "DATA", "AceType": "", "IsInherited": true }, { "PrincipalSID": "DATA", "PrincipalType": "DATA", "RightName": "DATA", "AceType": "", "IsInherited": true }, { "PrincipalSID": "DATA", "PrincipalType": "Unknown", "RightName": "DATA", "AceType": "", "IsInherited": true }, { "PrincipalSID": "DATA", "PrincipalType": "Group", "RightName": "GenericAll", "AceType": "", "IsInherited": true }, { "PrincipalSID": "DATA", "PrincipalType": "Unknown", "RightName": "GenericAll", "AceType": "", "IsInherited": true }, { "PrincipalSID": "DATA", "PrincipalType": "Group", "RightName": "WriteDacl", "AceType": "", "IsInherited": true }, { "PrincipalSID": "DATA", "PrincipalType": "Group", "RightName": "WriteOwner", "AceType": "", "IsInherited": true }, { "PrincipalSID": "DATA", "PrincipalType": "Group", "RightName": "GenericWrite", "AceType": "", "IsInherited": true } ] }, { "Properties": { "haslaps": false, "highvalue": false, "name": "DATA", "domain": "DATA", "objectid": "DATA", "distinguishedname": "DATA", "description": null, "enabled": true, "unconstraineddelegation": false, "serviceprincipalnames": [ "DATA", "DATA", "DATA", "DATA", "DATA", "DATA", "DATA", "DATA", "DATA", "DATA" ], "lastlogontimestamp": 1506599859, "pwdlastset": 1505682659, "operatingsystem": "DATA" }, "AllowedToDelegate": [], "AllowedToAct": [], "PrimaryGroupSid": "DATA", "Sessions": [], "LocalAdmins": [], "RemoteDesktopUsers": [], "DcomUsers": [], "PSRemoteUsers": [], "ObjectIdentifier": "DATA", "Aces": [ { "PrincipalSID": "DATA", "PrincipalType": "Group", "RightName": "Owner", "AceType": "", "IsInherited": false }, { "PrincipalSID": "DATA", "PrincipalType": "Group", "RightName": "GenericAll", "AceType": "", "IsInherited": false }, { "PrincipalSID": "DATA", "PrincipalType": "Group", "RightName": "GenericAll", "AceType": "", "IsInherited": false }, { "PrincipalSID": "DATA", "PrincipalType": "User", "RightName": "GenericAll", "AceType": "", "IsInherited": true }, { "PrincipalSID": "DATA", "PrincipalType": "Group", "RightName": "GenericAll", "AceType": "", "IsInherited": true }, { "PrincipalSID": "DATA", "PrincipalType": "Unknown", "RightName": "GenericAll", "AceType": "", "IsInherited": true }, { "PrincipalSID": "DATA", "PrincipalType": "Group", "RightName": "WriteDacl", "AceType": "", "IsInherited": true }, { "PrincipalSID": "DATA", "PrincipalType": "Group", "RightName": "WriteOwner", "AceType": "", "IsInherited": true }, { "PrincipalSID": "DATA", "PrincipalType": "Group", "RightName": "GenericWrite", "AceType": "", "IsInherited": true } ] <...truncated...>

Any suggestions on how I can get this to break properly & extract the field value pairs?  Thank you!

Labels (3)
Tags (2)
0 Karma
1 Solution

kamlesh_vaghela
SplunkTrust
SplunkTrust

@jason_hotchkiss 

Can you please try this?

[YOUR_SOURCETYPE]
SHOULD_LINEMERGE=false
LINE_BREAKER=]}(\,\s){\"Properties
NO_BINARY_CHECK=true
SEDCMD-a=s/{\"computers\": \[//g
SEDCMD-b=s/\]}\]}$/]}/g
TRUNCATE=0

 

Thanks
Kamlesh Vaghela

View solution in original post

kamlesh_vaghela
SplunkTrust
SplunkTrust

@jason_hotchkiss 

Can you please try this?

[YOUR_SOURCETYPE]
SHOULD_LINEMERGE=false
LINE_BREAKER=]}(\,\s){\"Properties
NO_BINARY_CHECK=true
SEDCMD-a=s/{\"computers\": \[//g
SEDCMD-b=s/\]}\]}$/]}/g
TRUNCATE=0

 

Thanks
Kamlesh Vaghela

jason_hotchkiss
Communicator

This works! Thank you!

0 Karma

jason_hotchkiss
Communicator

@kamlesh_vaghela 

Hello Kamlesh - that did not work, and I believe that was my fault, as the log format came in wrong.

Here is the regex:  https://regex101.com/r/KkpSIM/1

0 Karma

kamlesh_vaghela
SplunkTrust
SplunkTrust

@jason_hotchkiss 

 

Ca you please try this?

[ <SOURCETYPE NAME> ]
SHOULD_LINEMERGE=false
LINE_BREAKER=\]\n\s{4}}(,\s{5}){\n\s{6}"Properties":
NO_BINARY_CHECK=true
CHARSET=UTF-8
disabled=false
SEDCMD-a=s/{\n\s{2}"computers":\s\[\n\s{4}//g
SEDCMD-b=s/\n\s*//g
SEDCMD-c=s/\]}\]}$/]}/g

 

Your provided json was not valid. So I have added closing brackets at the end. 

Check the sample I have used.

https://regex101.com/r/NFxrJp/1

 

Thanks
KV ▄︻̷̿┻̿═━一

If this reply helps you, an upvote would be appreciated.

0 Karma
Get Updates on the Splunk Community!

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...