topic Updated: Help in event break for json file in Getting Data In

Updated: Help in event break for json file

AKG1_old1 — Wed, 06 Oct 2021 08:32:54 GMT

Hi,

Updated:

I am trying to break events which is in nested json. Each events start with { "links":

I have almost got it working. Just small part left is that now after each event there is one " , " and due to this event is not recognized as json event. Any idea how to remove it. Screenshot.

Props.conf

95% working props.
CHARSET = UTF-8
DATETIME_CONFIG =
KV_MODE = json
LINE_BREAKER = ([\r\n,]*(?:{[^[{]+\[)?){"links"
NO_BINARY_CHECK = true
SEDCMD-removefooter = s/(\]\,).*//g
SEDCMD-removeheader = s/\{\"data\": \[//g
SHOULD_LINEMERGE = false
TIME_FORMAT = %Y-%m-%dT%H:%M:%S.%3NZ
TIME_PREFIX = "endTime": "
TRUNCATE =
category = Custom
description = JavaScript Object Notation format. For more information, visit http://json.org/
disabled = false
pulldown_type = true

Sample data for 3 events ( each event starts with {"links": )

FYI: there is another 4th string {"links" which is extra value which I will remove using regex. basically consider data only in [ ]. all other will be removed using regex as its unnecessary.

{"data": [{"links": {"self": {"href": "/admin/jobs/81913"}, "file-lists": {"href": "https://test"}, "try-logs": {"href": "https://test"}}, "type": "job", "id": "81913", "attributes": {"jobId": 81913, "parentJobId": 0, "activeProcessId": 19776, "startTime": "2021-10-05T08:14:29.000Z", "endTime": "2021-10-05T08:14:53.000Z", "kilobytesDataTransferred": 0}}, {"links": {"self": {"href": "/admin/jobs/81912"}, "file-lists": {"href": "https://test"}, "try-logs": {"href": "https://test"}}, "type": "job", "id": "81912", "attributes": {"jobId": 81912, "parentJobId": 0,"startTime": "2021-10-05T08:14:04.000Z", "endTime": "2021-10-05T08:14:29.000Z", "jobQueueResource": "", "kilobytesDataTransferred": 0}}, {"links": {"self": {"href": "/admin/jobs/81911"}, "file-lists": {"href": "https://test"}, "try-logs": {"href": "https://test"}}, "type": "job", "id": "81911", "attributes": {"jobId": 81911, "parentJobId": 0, "startTime": "2021-10-05T05:44:01.000Z", "endTime": "2021-10-05T05:44:51.000Z", "kilobytesDataTransferred": 0}}], "meta": {"pagination": {"next": 10, "pages": 42, "last": 410, "offset": 0, "limit": 10, "count": 415, "page": 0, "first": 0}}, "links": {"next": {"href": "https://test"}, "self": {"href": "https://test"}, "last": {"href": "https://test"}, "first": {"href": "https://test"}}}

Thanks

Re: Help in event break for json file

danielcj — Tue, 05 Oct 2021 21:07:11 GMT

Hello,

Please, try the following (considering that everything before and after the [ and ] will be removed as you said):

[<sourcetype_name>] SHOULD_LINEMERGE=false LINE_BREAKER=(\,\s)\{\"links\"\: TIME_FORMAT=%Y-%m-%dT%H:%M:%S.%3NZ TIME_PREFIX=\"endTime\"\:\s\" MAX_TIMESTAMP_LOOKAHEAD=24

Re: Help in event break for json file

AKG1_old1 — Wed, 06 Oct 2021 08:29:06 GMT

Thanks but unfortunately it didn't work in my case. But I have almost working one. Just a small issue left.

After breaking events, each event has " , " at the end. This comma is causing that event is not recognize as json event. Any idea to fix it.

95% working props.
CHARSET = UTF-8
DATETIME_CONFIG =
KV_MODE = json
LINE_BREAKER = ([\r\n,]*(?:{[^[{]+\[)?){"links"
NO_BINARY_CHECK = true
SEDCMD-removefooter = s/(\]\,).*//g
SEDCMD-removeheader = s/\{\"data\": \[//g
SHOULD_LINEMERGE = false
TIME_FORMAT = %Y-%m-%dT%H:%M:%S.%3NZ
TIME_PREFIX = "endTime": "
TRUNCATE =
category = Custom
description = JavaScript Object Notation format. For more information, visit http://json.org/
disabled = false
pulldown_type = true

Re: Updated: Help in event break for json file

AKG1_old1 — Wed, 06 Oct 2021 15:40:47 GMT

Got it working
props.conf
CHARSET = UTF-8
DATETIME_CONFIG =
KV_MODE = json
LINE_BREAKER = ([\r\n,]*(?:{[^[{]+\[)?){.links.
NO_BINARY_CHECK = true
SHOULD_LINEMERGE = false
TIME_FORMAT = %Y-%m-%dT%H:%M:%S.%3NZ
TIME_PREFIX = .endTime.: .
TRUNCATE = 9999999
SEDCMD-replacequotes = s/'/"/g
SEDCMD-removecomma = s/,\s$//g
SEDCMD-removefooter = s/(\]\,).*//g
SEDCMD-removeheader = s/\{.data.: \[//g
category = Custom
description = JavaScript Object Notation format. For more information, visit http://json.org/
disabled = false
pulldown_type = true