Getting Data In

Line breaking not working for JSON logs (API at heavy forwarder).

asimasplunk
Explorer

We are using ingest pattern as API at Heavy forwarder.

props.conf:-

[kenna:applications]
INDEXED_EXTRACTIONS = json
TZ = UTC
LINE_BREAKER = "\}\,\{\"id\"\:
TRUNCATE = 10485760
SHOULD_LINEMERGE = false


This line breaker did not work

Sample Log:-

{"applications":[{"id":3964,"name":"xyz.com","repo_url":null,"host_name":null,"owner":null,"team_name":null,"business_units":null,"notes":null,"risk_meter_score":0,"vulnerability_count":0,"asset_count":0,"total_vulnerability_count":0,"open_vulnerability_count_by_risk_level":{"high":0,"medium":0,"low":0,"total":0},"historical_risk_meter_scores":[{"date":"2020-04-07","score":0},{"date":"2020-04-08","score":0},{"date":"2020-04-09","score":0},{"date":"2020-04-10","score":0},{"date":"2020-04-11","score":0},{"date":"2020-04-12","score":0},{"date":"2020-04-13","score":0},{"date":"2020-04-14","score":0},{"date":"2020-04-15","score":0},{"date":"2020-04-16","score":0},{"date":"2020-04-17","score":0},{"date":"2020-04-18","score":0},{"date":"2020-04-19","score":0},{"date":"2020-04-20","score":0},{"date":"2020-04-21","score":0},{"date":"2020-04-22","score":0},{"date":"2020-04-23","score":0},{"date":"2020-04-24","score":0},{"date":"2020-04-25","score":0},{"date":"2020-04-26","score":0},{"date":"2020-04-27","score":0},{"date":"2020-04-28","score":0},{"date":"2020-04-29","score":0},{"date":"2020-04-30","score":0},{"date":"2020-05-01","score":0},{"date":"2020-05-02","score":0},{"date":"2020-05-03","score":0},{"date":"2020-05-04","score":0},{"date":"2020-05-05","score":0},{"date":"2020-05-06","score":0},{"date":"2020-05-07","score":0},{"date":"2020-05-08","score":0},{"date":"2020-05-09","score":0},{"date":"2020-05-10","score":0},{"date":"2020-05-11","score":0},{"date":"2020-05-12","score":0},{"date":"2020-05-13","score":0},{"date":"2020-05-14","score":0},{"date":"2020-05-15","score":0},{"date":"2020-05-16","score":0},{"date":"2020-05-17","score":0},{"date":"2020-05-18","score":0},{"date":"2020-05-19","score":0},{"date":"2020-05-20","score":0},{"date":"2020-05-21","score":0},{"date":"2020-05-22","score":0},{"date":"2020-05-23","score":0},{"date":"2020-05-24","score":0},{"date":"2020-05-25","score":0},{"date":"2020-05-26","score":0},{"date":"2020-05-27","score":0},{"date":"2020-05-28","score":0},{"date":"2020-05-29","score":0},{"date":"2020-05-30","score":0},{"date":"2020-05-31","score":0},{"date":"2020-06-01","score":0},{"date":"2020-06-02","score":0},{"date":"2020-06-03","score":0},{"date":"2020-06-04","score":0},{"date":"2020-06-05","score":0},{"date":"2020-06-06","score":0},{"date":"2020-06-07","score":0},{"date":"2020-06-08","score":0},{"date":"2020-06-09","score":0},{"date":"2020-06-10","score":0},{"date":"2020-06-11","score":0},{"date":"2020-06-12","score":0},{"date":"2020-06-13","score":0},{"date":"2020-06-14","score":0},{"date":"2020-06-15","score":0},{"date":"2020-06-16","score":0},{"date":"2020-06-17","score":0},{"date":"2020-06-18","score":0},{"date":"2020-06-19","score":0},{"date":"2020-06-20","score":0},{"date":"2020-06-21","score":0},{"date":"2020-06-22","score":0},{"date":"2020-06-23","score":0},{"date":"2020-06-24","score":0},{"date":"2020-06-25","score":0},{"date":"2020-06-26","score":0},{"date":"2020-06-27","score":0},{"date":"2020-06-28","score":0},{"date":"2020-06-29","score":0},{"date":"2020-06-30","score":0},{"date":"2020-07-01","score":0},{"date":"2020-07-02","score":0},{"date":"2020-07-03","score":0},{"date":"2020-07-04","score":0},{"date":"2020-07-05","score":0},{"date":"2020-07-06","score":0}],"external_facing":true,"priority":10,"identifiers":["xyz.com"]},{"id":3965,"name":"xyz1.com/ecmlogin- DEV","repo_url":null,"host_name":null,"owner":null,"team_name":null,"business_units":null,"notes":null,"risk_meter_score":0,"vulnerability_count":0,"asset_count":0,"total_vulnerability_count":0,"open_vulnerability_count_by_risk_level":{"high":0,"medium":0,"low":0,"total":0},"historical_risk_meter_scores":[{"date":"2020-04-07","score":0},{"date":"2020-04-08","score":0},{"date":"2020-04-09","score":0},{"date":"2020-04-10","score":0},{..........

 

Labels (3)
Tags (1)
0 Karma
1 Solution

to4kawa
Ultra Champion

 

[kenna:applications]
TZ=UTC
KV_MODE=json
TRUNCATE=0
SHOULD_LINEMERGE=false
category=Structured
disabled=false
pulldown_type=true
LINE_BREAKER=(.){"id
SEDCMD-json=s/({.*})]}/\1/ s/.*applications.*//

 

LINE_BREAKER breaks JSON format. 
try  SEDCMD and KV_MODE=json

View solution in original post

richgalloway
SplunkTrust
SplunkTrust
Where do you expect/want the event to break?
There is no text matching '"},{"id":'.
---
If this reply helps you, Karma would be appreciated.

asimasplunk
Explorer

@richgalloway  Each log has a unique id. so we want to break every logs from },{"id":

0 Karma

richgalloway
SplunkTrust
SplunkTrust

Well then that is what should be the line breaker.  Also, every LINE_BREAKER setting must contain a capture group.  Try

LINE_BREAKER = }(,){"id":

 

---
If this reply helps you, Karma would be appreciated.
0 Karma

sumanssah
Communicator

Try something like 

 

LINE_BREAKER = {"id"

to4kawa
Ultra Champion

 

[kenna:applications]
TZ=UTC
KV_MODE=json
TRUNCATE=0
SHOULD_LINEMERGE=false
category=Structured
disabled=false
pulldown_type=true
LINE_BREAKER=(.){"id
SEDCMD-json=s/({.*})]}/\1/ s/.*applications.*//

 

LINE_BREAKER breaks JSON format. 
try  SEDCMD and KV_MODE=json

asimasplunk
Explorer

Thanks @to4kawa  this props.conf is working fine for my logs.... 😄

0 Karma
Got questions? Get answers!

Join the Splunk Community Slack to learn, troubleshoot, and make connections with fellow Splunk practitioners in real time!

Meet up IRL or virtually!

Join Splunk User Groups to connect and learn in-person by region or remotely by topic or industry.

Get Updates on the Splunk Community!

Enterprise Security Watch Party: What’s New in 8.8 - Recap

Splunk Community HQ hosted a Splunk Enterprise Security Watch Party: What’s New In ES 8.8 last Thursday. I had ...

Move CrowdStrike FDR ingestion to Data Inputs with a guided migration

CrowdStrike Falcon Data Replicator data is essential for security investigations, detections, and response. It ...

Make Your Machine Data AI-Ready in Splunk

Every team is getting ready for AI agents.    Watch On-Demand   In this Tech Talk, Splunk platform specialists ...