Line breaking not working for JSON logs (API at heavy forwarder).

asimasplunk — Mon, 06 Jul 2020 18:40:51 GMT

We are using ingest pattern as API at Heavy forwarder.

props.conf:-

[kenna:applications] INDEXED_EXTRACTIONS = json TZ = UTC LINE_BREAKER = "\}\,\{\"id\"\: TRUNCATE = 10485760 SHOULD_LINEMERGE = false

This line breaker did not work

Sample Log:-

{"applications":[{"id":3964,"name":"xyz.com","repo_url":null,"host_name":null,"owner":null,"team_name":null,"business_units":null,"notes":null,"risk_meter_score":0,"vulnerability_count":0,"asset_count":0,"total_vulnerability_count":0,"open_vulnerability_count_by_risk_level":{"high":0,"medium":0,"low":0,"total":0},"historical_risk_meter_scores":[{"date":"2020-04-07","score":0},{"date":"2020-04-08","score":0},{"date":"2020-04-09","score":0},{"date":"2020-04-10","score":0},{"date":"2020-04-11","score":0},{"date":"2020-04-12","score":0},{"date":"2020-04-13","score":0},{"date":"2020-04-14","score":0},{"date":"2020-04-15","score":0},{"date":"2020-04-16","score":0},{"date":"2020-04-17","score":0},{"date":"2020-04-18","score":0},{"date":"2020-04-19","score":0},{"date":"2020-04-20","score":0},{"date":"2020-04-21","score":0},{"date":"2020-04-22","score":0},{"date":"2020-04-23","score":0},{"date":"2020-04-24","score":0},{"date":"2020-04-25","score":0},{"date":"2020-04-26","score":0},{"date":"2020-04-27","score":0},{"date":"2020-04-28","score":0},{"date":"2020-04-29","score":0},{"date":"2020-04-30","score":0},{"date":"2020-05-01","score":0},{"date":"2020-05-02","score":0},{"date":"2020-05-03","score":0},{"date":"2020-05-04","score":0},{"date":"2020-05-05","score":0},{"date":"2020-05-06","score":0},{"date":"2020-05-07","score":0},{"date":"2020-05-08","score":0},{"date":"2020-05-09","score":0},{"date":"2020-05-10","score":0},{"date":"2020-05-11","score":0},{"date":"2020-05-12","score":0},{"date":"2020-05-13","score":0},{"date":"2020-05-14","score":0},{"date":"2020-05-15","score":0},{"date":"2020-05-16","score":0},{"date":"2020-05-17","score":0},{"date":"2020-05-18","score":0},{"date":"2020-05-19","score":0},{"date":"2020-05-20","score":0},{"date":"2020-05-21","score":0},{"date":"2020-05-22","score":0},{"date":"2020-05-23","score":0},{"date":"2020-05-24","score":0},{"date":"2020-05-25","score":0},{"date":"2020-05-26","score":0},{"date":"2020-05-27","score":0},{"date":"2020-05-28","score":0},{"date":"2020-05-29","score":0},{"date":"2020-05-30","score":0},{"date":"2020-05-31","score":0},{"date":"2020-06-01","score":0},{"date":"2020-06-02","score":0},{"date":"2020-06-03","score":0},{"date":"2020-06-04","score":0},{"date":"2020-06-05","score":0},{"date":"2020-06-06","score":0},{"date":"2020-06-07","score":0},{"date":"2020-06-08","score":0},{"date":"2020-06-09","score":0},{"date":"2020-06-10","score":0},{"date":"2020-06-11","score":0},{"date":"2020-06-12","score":0},{"date":"2020-06-13","score":0},{"date":"2020-06-14","score":0},{"date":"2020-06-15","score":0},{"date":"2020-06-16","score":0},{"date":"2020-06-17","score":0},{"date":"2020-06-18","score":0},{"date":"2020-06-19","score":0},{"date":"2020-06-20","score":0},{"date":"2020-06-21","score":0},{"date":"2020-06-22","score":0},{"date":"2020-06-23","score":0},{"date":"2020-06-24","score":0},{"date":"2020-06-25","score":0},{"date":"2020-06-26","score":0},{"date":"2020-06-27","score":0},{"date":"2020-06-28","score":0},{"date":"2020-06-29","score":0},{"date":"2020-06-30","score":0},{"date":"2020-07-01","score":0},{"date":"2020-07-02","score":0},{"date":"2020-07-03","score":0},{"date":"2020-07-04","score":0},{"date":"2020-07-05","score":0},{"date":"2020-07-06","score":0}],"external_facing":true,"priority":10,"identifiers":["xyz.com"]},{"id":3965,"name":"xyz1.com/ecmlogin- DEV","repo_url":null,"host_name":null,"owner":null,"team_name":null,"business_units":null,"notes":null,"risk_meter_score":0,"vulnerability_count":0,"asset_count":0,"total_vulnerability_count":0,"open_vulnerability_count_by_risk_level":{"high":0,"medium":0,"low":0,"total":0},"historical_risk_meter_scores":[{"date":"2020-04-07","score":0},{"date":"2020-04-08","score":0},{"date":"2020-04-09","score":0},{"date":"2020-04-10","score":0},{..........

Re: Line breaking not working for JSON logs

richgalloway — Mon, 06 Jul 2020 17:24:07 GMT

Where do you expect/want the event to break?
There is no text matching '"},{"id":'.

Re: Line breaking not working for JSON logs

asimasplunk — Mon, 06 Jul 2020 17:37:00 GMT

@richgalloway Each log has a unique id. so we want to break every logs from },{"id":

Re: Line breaking not working for JSON logs

richgalloway — Mon, 06 Jul 2020 17:59:22 GMT

Well then that is what should be the line breaker. Also, every LINE_BREAKER setting must contain a capture group. Try

LINE_BREAKER = }(,){"id":

Re: Line breaking not working for JSON logs

sumanssah — Thu, 16 Jul 2020 07:42:16 GMT

Try something like

LINE_BREAKER = {"id"

Re: Line breaking not working for JSON logs

to4kawa — Sat, 18 Jul 2020 02:42:21 GMT

[kenna:applications] TZ=UTC KV_MODE=json TRUNCATE=0 SHOULD_LINEMERGE=false category=Structured disabled=false pulldown_type=true LINE_BREAKER=(.){"id SEDCMD-json=s/({.*})]}/\1/ s/.*applications.*//

LINE_BREAKER breaks JSON format.
try SEDCMD and KV_MODE=json

Re: Line breaking not working for JSON logs

asimasplunk — Sat, 18 Jul 2020 14:44:21 GMT

Thanks @to4kawa this props.conf is working fine for my logs.... 😄

topic Re: Line breaking not working for JSON logs in Getting Data In

Line breaking not working for JSON logs (API at heavy forwarder).

Re: Line breaking not working for JSON logs

Re: Line breaking not working for JSON logs

Re: Line breaking not working for JSON logs

Re: Line breaking not working for JSON logs

Re: Line breaking not working for JSON logs

Re: Line breaking not working for JSON logs