Getting Data In

Updated: Help in event break for json file

AKG1_old1
Builder

Hi,

Updated:

I am trying to break events which is in nested json. Each events start with  { "links": 

I have almost got it working. Just small part left is that now after each event there is one " ,  "  and due to this event is not recognized as json event. Any idea how to remove it. Screenshot.

agoyal_0-1633509145601.png

 



Props.conf

95% working props.
CHARSET = UTF-8
DATETIME_CONFIG =
KV_MODE = json
LINE_BREAKER = ([\r\n,]*(?:{[^[{]+\[)?){"links"
NO_BINARY_CHECK = true
SEDCMD-removefooter = s/(\]\,).*//g
SEDCMD-removeheader = s/\{\"data\": \[//g
SHOULD_LINEMERGE = false
TIME_FORMAT = %Y-%m-%dT%H:%M:%S.%3NZ
TIME_PREFIX = "endTime": "
TRUNCATE =
category = Custom
description = JavaScript Object Notation format. For more information, visit http://json.org/
disabled = false
pulldown_type = true

Sample data for 3 events ( each event starts with {"links": ) 

FYI: there is another 4th string  {"links" which is extra value which I will remove using regex. basically consider data only in [ ]. all other will be removed using regex as its unnecessary.

{"data": [{"links": {"self": {"href": "/admin/jobs/81913"}, "file-lists": {"href": "https://test"}, "try-logs": {"href": "https://test"}}, "type": "job", "id": "81913", "attributes": {"jobId": 81913, "parentJobId": 0, "activeProcessId": 19776, "startTime": "2021-10-05T08:14:29.000Z", "endTime": "2021-10-05T08:14:53.000Z", "kilobytesDataTransferred": 0}}, {"links": {"self": {"href": "/admin/jobs/81912"}, "file-lists": {"href": "https://test"}, "try-logs": {"href": "https://test"}}, "type": "job", "id": "81912", "attributes": {"jobId": 81912, "parentJobId": 0,"startTime": "2021-10-05T08:14:04.000Z", "endTime": "2021-10-05T08:14:29.000Z", "jobQueueResource": "", "kilobytesDataTransferred": 0}}, {"links": {"self": {"href": "/admin/jobs/81911"}, "file-lists": {"href": "https://test"}, "try-logs": {"href": "https://test"}}, "type": "job", "id": "81911", "attributes": {"jobId": 81911, "parentJobId": 0, "startTime": "2021-10-05T05:44:01.000Z", "endTime": "2021-10-05T05:44:51.000Z", "kilobytesDataTransferred": 0}}], "meta": {"pagination": {"next": 10, "pages": 42, "last": 410, "offset": 0, "limit": 10, "count": 415, "page": 0, "first": 0}}, "links": {"next": {"href": "https://test"}, "self": {"href": "https://test"}, "last": {"href": "https://test"}, "first": {"href": "https://test"}}}

Thanks

Labels (2)
Tags (1)
0 Karma
1 Solution

AKG1_old1
Builder

Got it working 
props.conf
CHARSET = UTF-8
DATETIME_CONFIG =
KV_MODE = json
LINE_BREAKER = ([\r\n,]*(?:{[^[{]+\[)?){.links.
NO_BINARY_CHECK = true
SHOULD_LINEMERGE = false
TIME_FORMAT = %Y-%m-%dT%H:%M:%S.%3NZ
TIME_PREFIX = .endTime.: .
TRUNCATE = 9999999
SEDCMD-replacequotes = s/'/"/g
SEDCMD-removecomma = s/,\s$//g
SEDCMD-removefooter = s/(\]\,).*//g
SEDCMD-removeheader = s/\{.data.: \[//g
category = Custom
description = JavaScript Object Notation format. For more information, visit http://json.org/
disabled = false
pulldown_type = true

View solution in original post

0 Karma

AKG1_old1
Builder

Got it working 
props.conf
CHARSET = UTF-8
DATETIME_CONFIG =
KV_MODE = json
LINE_BREAKER = ([\r\n,]*(?:{[^[{]+\[)?){.links.
NO_BINARY_CHECK = true
SHOULD_LINEMERGE = false
TIME_FORMAT = %Y-%m-%dT%H:%M:%S.%3NZ
TIME_PREFIX = .endTime.: .
TRUNCATE = 9999999
SEDCMD-replacequotes = s/'/"/g
SEDCMD-removecomma = s/,\s$//g
SEDCMD-removefooter = s/(\]\,).*//g
SEDCMD-removeheader = s/\{.data.: \[//g
category = Custom
description = JavaScript Object Notation format. For more information, visit http://json.org/
disabled = false
pulldown_type = true

0 Karma

danielcj
Communicator

Hello,

 

Please, try the following (considering that everything before and after the [ and ] will be removed as you said):

 

 

[<sourcetype_name>]
SHOULD_LINEMERGE=false
LINE_BREAKER=(\,\s)\{\"links\"\:
TIME_FORMAT=%Y-%m-%dT%H:%M:%S.%3NZ
TIME_PREFIX=\"endTime\"\:\s\"
MAX_TIMESTAMP_LOOKAHEAD=24

 

0 Karma

AKG1_old1
Builder

Thanks but unfortunately it didn't work in my case. But I have almost working one. Just a small issue left.

After breaking events, each event has "  ,   "   at the end. This comma is causing that event is not recognize as json event. Any idea to fix it.


95% working props.
CHARSET = UTF-8
DATETIME_CONFIG =
KV_MODE = json
LINE_BREAKER = ([\r\n,]*(?:{[^[{]+\[)?){"links"
NO_BINARY_CHECK = true
SEDCMD-removefooter = s/(\]\,).*//g
SEDCMD-removeheader = s/\{\"data\": \[//g
SHOULD_LINEMERGE = false
TIME_FORMAT = %Y-%m-%dT%H:%M:%S.%3NZ
TIME_PREFIX = "endTime": "
TRUNCATE =
category = Custom
description = JavaScript Object Notation format. For more information, visit http://json.org/
disabled = false
pulldown_type = true

0 Karma
Get Updates on the Splunk Community!

Splunk Answers Content Calendar, June Edition

Get ready for this week’s post dedicated to Splunk Dashboards! We're celebrating the power of community by ...

What You Read The Most: Splunk Lantern’s Most Popular Articles!

Splunk Lantern is a Splunk customer success center that provides advice from Splunk experts on valuable data ...

See your relevant APM services, dashboards, and alerts in one place with the updated ...

As a Splunk Observability user, you have a lot of data you have to manage, prioritize, and troubleshoot on a ...