Getting Data In

Updated: Help in event break for json file

AKG1_old1
Builder

Hi,

Updated:

I am trying to break events which is in nested json. Each events start with  { "links": 

I have almost got it working. Just small part left is that now after each event there is one " ,  "  and due to this event is not recognized as json event. Any idea how to remove it. Screenshot.

agoyal_0-1633509145601.png

 



Props.conf

95% working props.
CHARSET = UTF-8
DATETIME_CONFIG =
KV_MODE = json
LINE_BREAKER = ([\r\n,]*(?:{[^[{]+\[)?){"links"
NO_BINARY_CHECK = true
SEDCMD-removefooter = s/(\]\,).*//g
SEDCMD-removeheader = s/\{\"data\": \[//g
SHOULD_LINEMERGE = false
TIME_FORMAT = %Y-%m-%dT%H:%M:%S.%3NZ
TIME_PREFIX = "endTime": "
TRUNCATE =
category = Custom
description = JavaScript Object Notation format. For more information, visit http://json.org/
disabled = false
pulldown_type = true

Sample data for 3 events ( each event starts with {"links": ) 

FYI: there is another 4th string  {"links" which is extra value which I will remove using regex. basically consider data only in [ ]. all other will be removed using regex as its unnecessary.

{"data": [{"links": {"self": {"href": "/admin/jobs/81913"}, "file-lists": {"href": "https://test"}, "try-logs": {"href": "https://test"}}, "type": "job", "id": "81913", "attributes": {"jobId": 81913, "parentJobId": 0, "activeProcessId": 19776, "startTime": "2021-10-05T08:14:29.000Z", "endTime": "2021-10-05T08:14:53.000Z", "kilobytesDataTransferred": 0}}, {"links": {"self": {"href": "/admin/jobs/81912"}, "file-lists": {"href": "https://test"}, "try-logs": {"href": "https://test"}}, "type": "job", "id": "81912", "attributes": {"jobId": 81912, "parentJobId": 0,"startTime": "2021-10-05T08:14:04.000Z", "endTime": "2021-10-05T08:14:29.000Z", "jobQueueResource": "", "kilobytesDataTransferred": 0}}, {"links": {"self": {"href": "/admin/jobs/81911"}, "file-lists": {"href": "https://test"}, "try-logs": {"href": "https://test"}}, "type": "job", "id": "81911", "attributes": {"jobId": 81911, "parentJobId": 0, "startTime": "2021-10-05T05:44:01.000Z", "endTime": "2021-10-05T05:44:51.000Z", "kilobytesDataTransferred": 0}}], "meta": {"pagination": {"next": 10, "pages": 42, "last": 410, "offset": 0, "limit": 10, "count": 415, "page": 0, "first": 0}}, "links": {"next": {"href": "https://test"}, "self": {"href": "https://test"}, "last": {"href": "https://test"}, "first": {"href": "https://test"}}}

Thanks

Labels (2)
Tags (1)
0 Karma
1 Solution

AKG1_old1
Builder

Got it working 
props.conf
CHARSET = UTF-8
DATETIME_CONFIG =
KV_MODE = json
LINE_BREAKER = ([\r\n,]*(?:{[^[{]+\[)?){.links.
NO_BINARY_CHECK = true
SHOULD_LINEMERGE = false
TIME_FORMAT = %Y-%m-%dT%H:%M:%S.%3NZ
TIME_PREFIX = .endTime.: .
TRUNCATE = 9999999
SEDCMD-replacequotes = s/'/"/g
SEDCMD-removecomma = s/,\s$//g
SEDCMD-removefooter = s/(\]\,).*//g
SEDCMD-removeheader = s/\{.data.: \[//g
category = Custom
description = JavaScript Object Notation format. For more information, visit http://json.org/
disabled = false
pulldown_type = true

View solution in original post

0 Karma

AKG1_old1
Builder

Got it working 
props.conf
CHARSET = UTF-8
DATETIME_CONFIG =
KV_MODE = json
LINE_BREAKER = ([\r\n,]*(?:{[^[{]+\[)?){.links.
NO_BINARY_CHECK = true
SHOULD_LINEMERGE = false
TIME_FORMAT = %Y-%m-%dT%H:%M:%S.%3NZ
TIME_PREFIX = .endTime.: .
TRUNCATE = 9999999
SEDCMD-replacequotes = s/'/"/g
SEDCMD-removecomma = s/,\s$//g
SEDCMD-removefooter = s/(\]\,).*//g
SEDCMD-removeheader = s/\{.data.: \[//g
category = Custom
description = JavaScript Object Notation format. For more information, visit http://json.org/
disabled = false
pulldown_type = true

0 Karma

danielcj
Communicator

Hello,

 

Please, try the following (considering that everything before and after the [ and ] will be removed as you said):

 

 

[<sourcetype_name>]
SHOULD_LINEMERGE=false
LINE_BREAKER=(\,\s)\{\"links\"\:
TIME_FORMAT=%Y-%m-%dT%H:%M:%S.%3NZ
TIME_PREFIX=\"endTime\"\:\s\"
MAX_TIMESTAMP_LOOKAHEAD=24

 

0 Karma

AKG1_old1
Builder

Thanks but unfortunately it didn't work in my case. But I have almost working one. Just a small issue left.

After breaking events, each event has "  ,   "   at the end. This comma is causing that event is not recognize as json event. Any idea to fix it.


95% working props.
CHARSET = UTF-8
DATETIME_CONFIG =
KV_MODE = json
LINE_BREAKER = ([\r\n,]*(?:{[^[{]+\[)?){"links"
NO_BINARY_CHECK = true
SEDCMD-removefooter = s/(\]\,).*//g
SEDCMD-removeheader = s/\{\"data\": \[//g
SHOULD_LINEMERGE = false
TIME_FORMAT = %Y-%m-%dT%H:%M:%S.%3NZ
TIME_PREFIX = "endTime": "
TRUNCATE =
category = Custom
description = JavaScript Object Notation format. For more information, visit http://json.org/
disabled = false
pulldown_type = true

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...