Updated: Help in event break for json file

[AKG1_old1]

Hi,

Updated:

I am trying to break events which is in nested json. Each events start with  { "links": 

I have almost got it working. Just small part left is that now after each event there is one " ,  "  and due to this event is not recognized as json event. Any idea how to remove it. Screenshot.

Props.conf

95% working props.
CHARSET = UTF-8
DATETIME_CONFIG =
KV_MODE = json
LINE_BREAKER = ([\r\n,]*(?:{[^[{]+\[)?){"links"
NO_BINARY_CHECK = true
SEDCMD-removefooter = s/(\]\,).*//g
SEDCMD-removeheader = s/\{\"data\": \[//g
SHOULD_LINEMERGE = false
TIME_FORMAT = %Y-%m-%dT%H:%M:%S.%3NZ
TIME_PREFIX = "endTime": "
TRUNCATE =
category = Custom
description = JavaScript Object Notation format. For more information, visit http://json.org/
disabled = false
pulldown_type = true

Sample data for 3 events ( each event starts with {"links": ) 

FYI: there is another 4th string  {"links" which is extra value which I will remove using regex. basically consider data only in [ ]. all other will be removed using regex as its unnecessary.

{"data": [{"links": {"self": {"href": "/admin/jobs/81913"}, "file-lists": {"href": "https://test"}, "try-logs": {"href": "https://test"}}, "type": "job", "id": "81913", "attributes": {"jobId": 81913, "parentJobId": 0, "activeProcessId": 19776, "startTime": "2021-10-05T08:14:29.000Z", "endTime": "2021-10-05T08:14:53.000Z", "kilobytesDataTransferred": 0}}, {"links": {"self": {"href": "/admin/jobs/81912"}, "file-lists": {"href": "https://test"}, "try-logs": {"href": "https://test"}}, "type": "job", "id": "81912", "attributes": {"jobId": 81912, "parentJobId": 0,"startTime": "2021-10-05T08:14:04.000Z", "endTime": "2021-10-05T08:14:29.000Z", "jobQueueResource": "", "kilobytesDataTransferred": 0}}, {"links": {"self": {"href": "/admin/jobs/81911"}, "file-lists": {"href": "https://test"}, "try-logs": {"href": "https://test"}}, "type": "job", "id": "81911", "attributes": {"jobId": 81911, "parentJobId": 0, "startTime": "2021-10-05T05:44:01.000Z", "endTime": "2021-10-05T05:44:51.000Z", "kilobytesDataTransferred": 0}}], "meta": {"pagination": {"next": 10, "pages": 42, "last": 410, "offset": 0, "limit": 10, "count": 415, "page": 0, "first": 0}}, "links": {"next": {"href": "https://test"}, "self": {"href": "https://test"}, "last": {"href": "https://test"}, "first": {"href": "https://test"}}}

Thanks

[AKG1_old1 · Accepted solution]

Got it working 
props.conf
CHARSET = UTF-8
DATETIME_CONFIG =
KV_MODE = json
LINE_BREAKER = ([\r\n,]*(?:{[^[{]+\[)?){.links.
NO_BINARY_CHECK = true
SHOULD_LINEMERGE = false
TIME_FORMAT = %Y-%m-%dT%H:%M:%S.%3NZ
TIME_PREFIX = .endTime.: .
TRUNCATE = 9999999
SEDCMD-replacequotes = s/'/"/g
SEDCMD-removecomma = s/,\s$//g
SEDCMD-removefooter = s/(\]\,).*//g
SEDCMD-removeheader = s/\{.data.: \[//g
category = Custom
description = JavaScript Object Notation format. For more information, visit http://json.org/
disabled = false
pulldown_type = true

[danielcj]

Hello,

Please, try the following (considering that everything before and after the [ and ] will be removed as you said):

[<sourcetype_name>]
SHOULD_LINEMERGE=false
LINE_BREAKER=(\,\s)\{\"links\"\:
TIME_FORMAT=%Y-%m-%dT%H:%M:%S.%3NZ
TIME_PREFIX=\"endTime\"\:\s\"
MAX_TIMESTAMP_LOOKAHEAD=24

[AKG1_old1]

Thanks but unfortunately it didn't work in my case. But I have almost working one. Just a small issue left.

After breaking events, each event has "  ,   "   at the end. This comma is causing that event is not recognize as json event. Any idea to fix it.

95% working props.
CHARSET = UTF-8
DATETIME_CONFIG =
KV_MODE = json
LINE_BREAKER = ([\r\n,]*(?:{[^[{]+\[)?){"links"
NO_BINARY_CHECK = true
SEDCMD-removefooter = s/(\]\,).*//g
SEDCMD-removeheader = s/\{\"data\": \[//g
SHOULD_LINEMERGE = false
TIME_FORMAT = %Y-%m-%dT%H:%M:%S.%3NZ
TIME_PREFIX = "endTime": "
TRUNCATE =
category = Custom
description = JavaScript Object Notation format. For more information, visit http://json.org/
disabled = false
pulldown_type = true