Getting Data In

Line breaking not working for JSON logs (API at heavy forwarder).

asimasplunk
Explorer

We are using ingest pattern as API at Heavy forwarder.

props.conf:-

[kenna:applications]
INDEXED_EXTRACTIONS = json
TZ = UTC
LINE_BREAKER = "\}\,\{\"id\"\:
TRUNCATE = 10485760
SHOULD_LINEMERGE = false


This line breaker did not work

Sample Log:-

{"applications":[{"id":3964,"name":"xyz.com","repo_url":null,"host_name":null,"owner":null,"team_name":null,"business_units":null,"notes":null,"risk_meter_score":0,"vulnerability_count":0,"asset_count":0,"total_vulnerability_count":0,"open_vulnerability_count_by_risk_level":{"high":0,"medium":0,"low":0,"total":0},"historical_risk_meter_scores":[{"date":"2020-04-07","score":0},{"date":"2020-04-08","score":0},{"date":"2020-04-09","score":0},{"date":"2020-04-10","score":0},{"date":"2020-04-11","score":0},{"date":"2020-04-12","score":0},{"date":"2020-04-13","score":0},{"date":"2020-04-14","score":0},{"date":"2020-04-15","score":0},{"date":"2020-04-16","score":0},{"date":"2020-04-17","score":0},{"date":"2020-04-18","score":0},{"date":"2020-04-19","score":0},{"date":"2020-04-20","score":0},{"date":"2020-04-21","score":0},{"date":"2020-04-22","score":0},{"date":"2020-04-23","score":0},{"date":"2020-04-24","score":0},{"date":"2020-04-25","score":0},{"date":"2020-04-26","score":0},{"date":"2020-04-27","score":0},{"date":"2020-04-28","score":0},{"date":"2020-04-29","score":0},{"date":"2020-04-30","score":0},{"date":"2020-05-01","score":0},{"date":"2020-05-02","score":0},{"date":"2020-05-03","score":0},{"date":"2020-05-04","score":0},{"date":"2020-05-05","score":0},{"date":"2020-05-06","score":0},{"date":"2020-05-07","score":0},{"date":"2020-05-08","score":0},{"date":"2020-05-09","score":0},{"date":"2020-05-10","score":0},{"date":"2020-05-11","score":0},{"date":"2020-05-12","score":0},{"date":"2020-05-13","score":0},{"date":"2020-05-14","score":0},{"date":"2020-05-15","score":0},{"date":"2020-05-16","score":0},{"date":"2020-05-17","score":0},{"date":"2020-05-18","score":0},{"date":"2020-05-19","score":0},{"date":"2020-05-20","score":0},{"date":"2020-05-21","score":0},{"date":"2020-05-22","score":0},{"date":"2020-05-23","score":0},{"date":"2020-05-24","score":0},{"date":"2020-05-25","score":0},{"date":"2020-05-26","score":0},{"date":"2020-05-27","score":0},{"date":"2020-05-28","score":0},{"date":"2020-05-29","score":0},{"date":"2020-05-30","score":0},{"date":"2020-05-31","score":0},{"date":"2020-06-01","score":0},{"date":"2020-06-02","score":0},{"date":"2020-06-03","score":0},{"date":"2020-06-04","score":0},{"date":"2020-06-05","score":0},{"date":"2020-06-06","score":0},{"date":"2020-06-07","score":0},{"date":"2020-06-08","score":0},{"date":"2020-06-09","score":0},{"date":"2020-06-10","score":0},{"date":"2020-06-11","score":0},{"date":"2020-06-12","score":0},{"date":"2020-06-13","score":0},{"date":"2020-06-14","score":0},{"date":"2020-06-15","score":0},{"date":"2020-06-16","score":0},{"date":"2020-06-17","score":0},{"date":"2020-06-18","score":0},{"date":"2020-06-19","score":0},{"date":"2020-06-20","score":0},{"date":"2020-06-21","score":0},{"date":"2020-06-22","score":0},{"date":"2020-06-23","score":0},{"date":"2020-06-24","score":0},{"date":"2020-06-25","score":0},{"date":"2020-06-26","score":0},{"date":"2020-06-27","score":0},{"date":"2020-06-28","score":0},{"date":"2020-06-29","score":0},{"date":"2020-06-30","score":0},{"date":"2020-07-01","score":0},{"date":"2020-07-02","score":0},{"date":"2020-07-03","score":0},{"date":"2020-07-04","score":0},{"date":"2020-07-05","score":0},{"date":"2020-07-06","score":0}],"external_facing":true,"priority":10,"identifiers":["xyz.com"]},{"id":3965,"name":"xyz1.com/ecmlogin- DEV","repo_url":null,"host_name":null,"owner":null,"team_name":null,"business_units":null,"notes":null,"risk_meter_score":0,"vulnerability_count":0,"asset_count":0,"total_vulnerability_count":0,"open_vulnerability_count_by_risk_level":{"high":0,"medium":0,"low":0,"total":0},"historical_risk_meter_scores":[{"date":"2020-04-07","score":0},{"date":"2020-04-08","score":0},{"date":"2020-04-09","score":0},{"date":"2020-04-10","score":0},{..........

 

Labels (3)
Tags (1)
0 Karma
1 Solution

to4kawa
Ultra Champion

 

[kenna:applications]
TZ=UTC
KV_MODE=json
TRUNCATE=0
SHOULD_LINEMERGE=false
category=Structured
disabled=false
pulldown_type=true
LINE_BREAKER=(.){"id
SEDCMD-json=s/({.*})]}/\1/ s/.*applications.*//

 

LINE_BREAKER breaks JSON format. 
try  SEDCMD and KV_MODE=json

View solution in original post

richgalloway
SplunkTrust
SplunkTrust
Where do you expect/want the event to break?
There is no text matching '"},{"id":'.
---
If this reply helps you, Karma would be appreciated.

asimasplunk
Explorer

@richgalloway  Each log has a unique id. so we want to break every logs from },{"id":

0 Karma

richgalloway
SplunkTrust
SplunkTrust

Well then that is what should be the line breaker.  Also, every LINE_BREAKER setting must contain a capture group.  Try

LINE_BREAKER = }(,){"id":

 

---
If this reply helps you, Karma would be appreciated.
0 Karma

sumanssah
Communicator

Try something like 

 

LINE_BREAKER = {"id"

to4kawa
Ultra Champion

 

[kenna:applications]
TZ=UTC
KV_MODE=json
TRUNCATE=0
SHOULD_LINEMERGE=false
category=Structured
disabled=false
pulldown_type=true
LINE_BREAKER=(.){"id
SEDCMD-json=s/({.*})]}/\1/ s/.*applications.*//

 

LINE_BREAKER breaks JSON format. 
try  SEDCMD and KV_MODE=json

asimasplunk
Explorer

Thanks @to4kawa  this props.conf is working fine for my logs.... 😄

0 Karma
Get Updates on the Splunk Community!

Splunk Answers Content Calendar, June Edition

Get ready for this week’s post dedicated to Splunk Dashboards! We're celebrating the power of community by ...

What You Read The Most: Splunk Lantern’s Most Popular Articles!

Splunk Lantern is a Splunk customer success center that provides advice from Splunk experts on valuable data ...

See your relevant APM services, dashboards, and alerts in one place with the updated ...

As a Splunk Observability user, you have a lot of data you have to manage, prioritize, and troubleshoot on a ...