Getting Data In

Why is the event line breaking not working properly for JSON format?

fmorar
Engager

I tried a few solutions but none worked for me so far:

answer-614348

I have an application that writes in some sort of json format, problem is that many events show up within the same second.
I am trying to break the line without successs:

{"flow_id":"kiozeybzkmg","correlation_id":"d5589ebc-d17a-4b0e-b7f1-2e754c20e009","tenant_code":"VAULTTESTTENANT","fusion_prod_code":"VAULTTEST","event_name":"ECSDeposit","event_timestamp":"01/29/2018 10:22:41.649","duration":0.0,"request_url":"http://localsite:61018/va/contact.svc/econtacts","machine_name":"HostD002","method_name":"DT.Service.EcontractVault.HttpListener.EcontractVault.SignbindAndDepositContract","title":"Econ Vault Operations","message":"New request arrived","tran_info":{"deal_id":"835514948","functional_area":"ECONAFFIX","dealer_code":141348,"user_code":362266,"lender_id":"DTL","request_type":"Signing"},"extended_properties":{"IP":"10.133.148.159|::1"}}
{"flow_id":"sp0pn1aaari","correlation_id":"9d627546-05e3-45c5-aa08-3ec6800266f4","tenant_code":"VAULTTESTTENANT","fusion_prod_code":"VAULTTEST","event_name":"ECSDeposit","event_timestamp":"01/29/2018 10:22:41.897","duration":3106.2043000000003,"request_url":"http://localsite:61018/va/contact.svc/econtacts","machine_name":"HostD002","method_name":"","title":"","message":"Request processed successfully","tran_info":{"deal_id":"816881309","functional_area":"ECONAFFIX","dealer_code":141348,"user_code":362266,"lender_id":"DTL","dealer_id":"146387","request_type":"Signing"},"extended_properties":{"IP":"10.133.148.159|::1"}}
{"flow_id":"k302hfrxcze","correlation_id":"0be872d8-ea2d-43b1-b29b-f7908ecd664f","tenant_code":"VAULTTESTTENANT","fusion_prod_code":"VAULTTEST","event_name":"ECSRetrieve","event_timestamp":"01/29/2018 10:22:43.607","duration":0.0,"request_url":"http://localsite:61018/va/contact.svc/econtacts/1052368321/66909516/66909518/cc","machine_name":"HostD002","method_name":"DT.Service.EcontractVault.HttpListener.EcontractVault.RetrieveContract","title":"Econ Vault Operations","message":"New request arrived","tran_info":{"deal_id":"1052368321","functional_area":"ECONAFFIX","dealer_code":141348,"user_code":362266,"request_type":"CC"},"extended_properties":{"IP":"10.133.148.159|::1","TranSid":"66909516","ProfSid":"66909518"}}
{"flow_id":"2xxv2neh4f1","correlation_id":"21dcd29f-dc4f-4c6e-8b9e-993c4ce47971","tenant_code":"VAULTTESTTENANT","fusion_prod_code":"VAULTTEST","event_name":"ECSDeposit","event_timestamp":"01/29/2018 10:22:43.622","duration":0.0,"request_url":"http://localsite:61018/va/contact.svc/econtacts","machine_name":"HostD002","method_name":"DT.Service.EcontractVault.HttpListener.EcontractVault.SignbindAndDepositContract","title":"Econ Vault Operations","message":"New request arrived","tran_info":{"deal_id":"828082541","functional_area":"ECONAFFIX","dealer_code":141348,"user_code":362266,"lender_id":"DTL","request_type":"Signing"},"extended_properties":{"IP":"10.133.148.159|::1"}}
{"flow_id":"kiozeybzkmg","correlation_id":"d5589ebc-d17a-4b0e-b7f1-2e754c20e009","tenant_code":"VAULTTESTTENANT","fusion_prod_code":"VAULTTEST","event_name":"ECSDeposit","event_timestamp":"01/29/2018 10:22:44.595","duration":2947.1757000000002,"request_url":"http://localsite:61018/va/contact.svc/econtacts","machine_name":"HostD002","method_name":"","title":"","message":"Request processed successfully","tran_info":{"deal_id":"835514948","functional_area":"ECONAFFIX","dealer_code":141348,"user_code":362266,"lender_id":"DTL","dealer_id":"146387","request_type":"Signing"},"extended_properties":{"IP":"10.133.148.159|::1"}}

This is what I have in my transforms.conf:

 KV_MODE = json
 LINE_BREAKER = ([\r\n]+)(?=\s*\{\s*\"flow_id\")
 #TRUNCATE = 0
 SHOULD_LINEMERGE=false
 NO_BINARY_CHECK=true
 TIME_FORMAT= \"%m/%d/%Y %H:%M:%S.%3N\"
 TIME_PREFIX = \s*\"event_timestamp\"
 MAX_TIMESTAMP_LOOKAHEAD=26
 #BREAK_ONLY_BEFORE = \{\s*\"flow_id\"
0 Karma

apoorvaaj
Engager

Hey, @mayurr98 is right, you should save those settings in props.conf and not in transforms.conf

Also, I slightly updated the time properties :

[ sourcetype]
CHARSET=UTF-8
KV_MODE=JSON
NO_BINARY_CHECK=true
SHOULD_LINEMERGE=false
TIME_FORMAT=%m/%d/%Y %H:%M:%S.%3N
TIME_PREFIX="event_timestamp":"
disabled=false
pulldown_type=true

0 Karma

mayurr98
Super Champion

hey, I ingested above data in with the following configuration and it worked for me. And you have to write this in props.conf and NOT transforms.conf

[<sourcetype>]
DATETIME_CONFIG =
KV_MODE = json
MAX_TIMESTAMP_LOOKAHEAD = 26
NO_BINARY_CHECK = true
SHOULD_LINEMERGE = false
TIME_FORMAT = %m/%d/%Y %H:%M:%S.%3N
TIME_PREFIX = event_timestamp\":\"
category = Custom
pulldown_type = true

let me know if this helps!

0 Karma
Get Updates on the Splunk Community!

What's New in Splunk Cloud Platform 9.2.2403?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.2.2403! Analysts can ...

Stay Connected: Your Guide to July and August Tech Talks, Office Hours, and Webinars!

Dive into our sizzling summer lineup for July and August Community Office Hours and Tech Talks. Scroll down to ...

Edge Processor Scaling, Energy & Manufacturing Use Cases, and More New Articles on ...

Splunk Lantern is a Splunk customer success center that provides advice from Splunk experts on valuable data ...