Line breaking not working for JSON logs (API at heavy forwarder).

[asimasplunk]

We are using ingest pattern as API at Heavy forwarder.

props.conf:-

[kenna:applications]
INDEXED_EXTRACTIONS = json
TZ = UTC
LINE_BREAKER = "\}\,\{\"id\"\:
TRUNCATE = 10485760
SHOULD_LINEMERGE = false

This line breaker did not work

Sample Log:-

{"applications":[{"id":3964,"name":"xyz.com","repo_url":null,"host_name":null,"owner":null,"team_name":null,"business_units":null,"notes":null,"risk_meter_score":0,"vulnerability_count":0,"asset_count":0,"total_vulnerability_count":0,"open_vulnerability_count_by_risk_level":{"high":0,"medium":0,"low":0,"total":0},"historical_risk_meter_scores":[{"date":"2020-04-07","score":0},{"date":"2020-04-08","score":0},{"date":"2020-04-09","score":0},{"date":"2020-04-10","score":0},{"date":"2020-04-11","score":0},{"date":"2020-04-12","score":0},{"date":"2020-04-13","score":0},{"date":"2020-04-14","score":0},{"date":"2020-04-15","score":0},{"date":"2020-04-16","score":0},{"date":"2020-04-17","score":0},{"date":"2020-04-18","score":0},{"date":"2020-04-19","score":0},{"date":"2020-04-20","score":0},{"date":"2020-04-21","score":0},{"date":"2020-04-22","score":0},{"date":"2020-04-23","score":0},{"date":"2020-04-24","score":0},{"date":"2020-04-25","score":0},{"date":"2020-04-26","score":0},{"date":"2020-04-27","score":0},{"date":"2020-04-28","score":0},{"date":"2020-04-29","score":0},{"date":"2020-04-30","score":0},{"date":"2020-05-01","score":0},{"date":"2020-05-02","score":0},{"date":"2020-05-03","score":0},{"date":"2020-05-04","score":0},{"date":"2020-05-05","score":0},{"date":"2020-05-06","score":0},{"date":"2020-05-07","score":0},{"date":"2020-05-08","score":0},{"date":"2020-05-09","score":0},{"date":"2020-05-10","score":0},{"date":"2020-05-11","score":0},{"date":"2020-05-12","score":0},{"date":"2020-05-13","score":0},{"date":"2020-05-14","score":0},{"date":"2020-05-15","score":0},{"date":"2020-05-16","score":0},{"date":"2020-05-17","score":0},{"date":"2020-05-18","score":0},{"date":"2020-05-19","score":0},{"date":"2020-05-20","score":0},{"date":"2020-05-21","score":0},{"date":"2020-05-22","score":0},{"date":"2020-05-23","score":0},{"date":"2020-05-24","score":0},{"date":"2020-05-25","score":0},{"date":"2020-05-26","score":0},{"date":"2020-05-27","score":0},{"date":"2020-05-28","score":0},{"date":"2020-05-29","score":0},{"date":"2020-05-30","score":0},{"date":"2020-05-31","score":0},{"date":"2020-06-01","score":0},{"date":"2020-06-02","score":0},{"date":"2020-06-03","score":0},{"date":"2020-06-04","score":0},{"date":"2020-06-05","score":0},{"date":"2020-06-06","score":0},{"date":"2020-06-07","score":0},{"date":"2020-06-08","score":0},{"date":"2020-06-09","score":0},{"date":"2020-06-10","score":0},{"date":"2020-06-11","score":0},{"date":"2020-06-12","score":0},{"date":"2020-06-13","score":0},{"date":"2020-06-14","score":0},{"date":"2020-06-15","score":0},{"date":"2020-06-16","score":0},{"date":"2020-06-17","score":0},{"date":"2020-06-18","score":0},{"date":"2020-06-19","score":0},{"date":"2020-06-20","score":0},{"date":"2020-06-21","score":0},{"date":"2020-06-22","score":0},{"date":"2020-06-23","score":0},{"date":"2020-06-24","score":0},{"date":"2020-06-25","score":0},{"date":"2020-06-26","score":0},{"date":"2020-06-27","score":0},{"date":"2020-06-28","score":0},{"date":"2020-06-29","score":0},{"date":"2020-06-30","score":0},{"date":"2020-07-01","score":0},{"date":"2020-07-02","score":0},{"date":"2020-07-03","score":0},{"date":"2020-07-04","score":0},{"date":"2020-07-05","score":0},{"date":"2020-07-06","score":0}],"external_facing":true,"priority":10,"identifiers":["xyz.com"]},{"id":3965,"name":"xyz1.com/ecmlogin- DEV","repo_url":null,"host_name":null,"owner":null,"team_name":null,"business_units":null,"notes":null,"risk_meter_score":0,"vulnerability_count":0,"asset_count":0,"total_vulnerability_count":0,"open_vulnerability_count_by_risk_level":{"high":0,"medium":0,"low":0,"total":0},"historical_risk_meter_scores":[{"date":"2020-04-07","score":0},{"date":"2020-04-08","score":0},{"date":"2020-04-09","score":0},{"date":"2020-04-10","score":0},{..........

[richgalloway]

Where do you expect/want the event to break?
There is no text matching '"},{"id":'.

[asimasplunk]

@richgalloway  Each log has a unique id. so we want to break every logs from },{"id":

[richgalloway]

Well then that is what should be the line breaker.  Also, every LINE_BREAKER setting must contain a capture group.  Try

LINE_BREAKER = }(,){"id":

[sumanssah]

Try something like 

LINE_BREAKER = {"id"

[to4kawa · Accepted solution]

[kenna:applications]
TZ=UTC
KV_MODE=json
TRUNCATE=0
SHOULD_LINEMERGE=false
category=Structured
disabled=false
pulldown_type=true
LINE_BREAKER=(.){"id
SEDCMD-json=s/({.*})]}/\1/ s/.*applications.*//

LINE_BREAKER breaks JSON format. 
try  SEDCMD and KV_MODE=json

[asimasplunk]

Thanks @to4kawa  this props.conf is working fine for my logs.... 😄