Getting Data In

Updated: Help in event break for json file

AKG1_old1
Builder

Hi,

Updated:

I am trying to break events which is in nested json. Each events start with  { "links": 

I have almost got it working. Just small part left is that now after each event there is one " ,  "  and due to this event is not recognized as json event. Any idea how to remove it. Screenshot.

agoyal_0-1633509145601.png

 



Props.conf

95% working props.
CHARSET = UTF-8
DATETIME_CONFIG =
KV_MODE = json
LINE_BREAKER = ([\r\n,]*(?:{[^[{]+\[)?){"links"
NO_BINARY_CHECK = true
SEDCMD-removefooter = s/(\]\,).*//g
SEDCMD-removeheader = s/\{\"data\": \[//g
SHOULD_LINEMERGE = false
TIME_FORMAT = %Y-%m-%dT%H:%M:%S.%3NZ
TIME_PREFIX = "endTime": "
TRUNCATE =
category = Custom
description = JavaScript Object Notation format. For more information, visit http://json.org/
disabled = false
pulldown_type = true

Sample data for 3 events ( each event starts with {"links": ) 

FYI: there is another 4th string  {"links" which is extra value which I will remove using regex. basically consider data only in [ ]. all other will be removed using regex as its unnecessary.

{"data": [{"links": {"self": {"href": "/admin/jobs/81913"}, "file-lists": {"href": "https://test"}, "try-logs": {"href": "https://test"}}, "type": "job", "id": "81913", "attributes": {"jobId": 81913, "parentJobId": 0, "activeProcessId": 19776, "startTime": "2021-10-05T08:14:29.000Z", "endTime": "2021-10-05T08:14:53.000Z", "kilobytesDataTransferred": 0}}, {"links": {"self": {"href": "/admin/jobs/81912"}, "file-lists": {"href": "https://test"}, "try-logs": {"href": "https://test"}}, "type": "job", "id": "81912", "attributes": {"jobId": 81912, "parentJobId": 0,"startTime": "2021-10-05T08:14:04.000Z", "endTime": "2021-10-05T08:14:29.000Z", "jobQueueResource": "", "kilobytesDataTransferred": 0}}, {"links": {"self": {"href": "/admin/jobs/81911"}, "file-lists": {"href": "https://test"}, "try-logs": {"href": "https://test"}}, "type": "job", "id": "81911", "attributes": {"jobId": 81911, "parentJobId": 0, "startTime": "2021-10-05T05:44:01.000Z", "endTime": "2021-10-05T05:44:51.000Z", "kilobytesDataTransferred": 0}}], "meta": {"pagination": {"next": 10, "pages": 42, "last": 410, "offset": 0, "limit": 10, "count": 415, "page": 0, "first": 0}}, "links": {"next": {"href": "https://test"}, "self": {"href": "https://test"}, "last": {"href": "https://test"}, "first": {"href": "https://test"}}}

Thanks

Labels (2)
Tags (1)
0 Karma
1 Solution

AKG1_old1
Builder

Got it working 
props.conf
CHARSET = UTF-8
DATETIME_CONFIG =
KV_MODE = json
LINE_BREAKER = ([\r\n,]*(?:{[^[{]+\[)?){.links.
NO_BINARY_CHECK = true
SHOULD_LINEMERGE = false
TIME_FORMAT = %Y-%m-%dT%H:%M:%S.%3NZ
TIME_PREFIX = .endTime.: .
TRUNCATE = 9999999
SEDCMD-replacequotes = s/'/"/g
SEDCMD-removecomma = s/,\s$//g
SEDCMD-removefooter = s/(\]\,).*//g
SEDCMD-removeheader = s/\{.data.: \[//g
category = Custom
description = JavaScript Object Notation format. For more information, visit http://json.org/
disabled = false
pulldown_type = true

View solution in original post

0 Karma

AKG1_old1
Builder

Got it working 
props.conf
CHARSET = UTF-8
DATETIME_CONFIG =
KV_MODE = json
LINE_BREAKER = ([\r\n,]*(?:{[^[{]+\[)?){.links.
NO_BINARY_CHECK = true
SHOULD_LINEMERGE = false
TIME_FORMAT = %Y-%m-%dT%H:%M:%S.%3NZ
TIME_PREFIX = .endTime.: .
TRUNCATE = 9999999
SEDCMD-replacequotes = s/'/"/g
SEDCMD-removecomma = s/,\s$//g
SEDCMD-removefooter = s/(\]\,).*//g
SEDCMD-removeheader = s/\{.data.: \[//g
category = Custom
description = JavaScript Object Notation format. For more information, visit http://json.org/
disabled = false
pulldown_type = true

0 Karma

danielcj
Communicator

Hello,

 

Please, try the following (considering that everything before and after the [ and ] will be removed as you said):

 

 

[<sourcetype_name>]
SHOULD_LINEMERGE=false
LINE_BREAKER=(\,\s)\{\"links\"\:
TIME_FORMAT=%Y-%m-%dT%H:%M:%S.%3NZ
TIME_PREFIX=\"endTime\"\:\s\"
MAX_TIMESTAMP_LOOKAHEAD=24

 

0 Karma

AKG1_old1
Builder

Thanks but unfortunately it didn't work in my case. But I have almost working one. Just a small issue left.

After breaking events, each event has "  ,   "   at the end. This comma is causing that event is not recognize as json event. Any idea to fix it.


95% working props.
CHARSET = UTF-8
DATETIME_CONFIG =
KV_MODE = json
LINE_BREAKER = ([\r\n,]*(?:{[^[{]+\[)?){"links"
NO_BINARY_CHECK = true
SEDCMD-removefooter = s/(\]\,).*//g
SEDCMD-removeheader = s/\{\"data\": \[//g
SHOULD_LINEMERGE = false
TIME_FORMAT = %Y-%m-%dT%H:%M:%S.%3NZ
TIME_PREFIX = "endTime": "
TRUNCATE =
category = Custom
description = JavaScript Object Notation format. For more information, visit http://json.org/
disabled = false
pulldown_type = true

0 Karma
Get Updates on the Splunk Community!

Admin Your Splunk Cloud, Your Way

Join us to maximize different techniques to best tune Splunk Cloud. In this Tech Enablement, you will get ...

Cloud Platform | Discontinuing support for TLS version 1.0 and 1.1

Overview Transport Layer Security (TLS) is a security communications protocol that lets two computers, ...

New Customer Testimonials

Enterprises of all sizes and across different industries are accelerating cloud adoption by migrating ...