Why is the event line breaking not working properly for JSON format?

fmorar — Mon, 29 Jan 2018 20:11:26 GMT

I tried a few solutions but none worked for me so far:

answer-614348

I have an application that writes in some sort of json format, problem is that many events show up within the same second.
I am trying to break the line without successs:

{"flow_id":"kiozeybzkmg","correlation_id":"d5589ebc-d17a-4b0e-b7f1-2e754c20e009","tenant_code":"VAULTTESTTENANT","fusion_prod_code":"VAULTTEST","event_name":"ECSDeposit","event_timestamp":"01/29/2018 10:22:41.649","duration":0.0,"request_url":"http://localsite:61018/va/contact.svc/econtacts","machine_name":"HostD002","method_name":"DT.Service.EcontractVault.HttpListener.EcontractVault.SignbindAndDepositContract","title":"Econ Vault Operations","message":"New request arrived","tran_info":{"deal_id":"835514948","functional_area":"ECONAFFIX","dealer_code":141348,"user_code":362266,"lender_id":"DTL","request_type":"Signing"},"extended_properties":{"IP":"10.133.148.159|::1"}}
{"flow_id":"sp0pn1aaari","correlation_id":"9d627546-05e3-45c5-aa08-3ec6800266f4","tenant_code":"VAULTTESTTENANT","fusion_prod_code":"VAULTTEST","event_name":"ECSDeposit","event_timestamp":"01/29/2018 10:22:41.897","duration":3106.2043000000003,"request_url":"http://localsite:61018/va/contact.svc/econtacts","machine_name":"HostD002","method_name":"","title":"","message":"Request processed successfully","tran_info":{"deal_id":"816881309","functional_area":"ECONAFFIX","dealer_code":141348,"user_code":362266,"lender_id":"DTL","dealer_id":"146387","request_type":"Signing"},"extended_properties":{"IP":"10.133.148.159|::1"}}
{"flow_id":"k302hfrxcze","correlation_id":"0be872d8-ea2d-43b1-b29b-f7908ecd664f","tenant_code":"VAULTTESTTENANT","fusion_prod_code":"VAULTTEST","event_name":"ECSRetrieve","event_timestamp":"01/29/2018 10:22:43.607","duration":0.0,"request_url":"http://localsite:61018/va/contact.svc/econtacts/1052368321/66909516/66909518/cc","machine_name":"HostD002","method_name":"DT.Service.EcontractVault.HttpListener.EcontractVault.RetrieveContract","title":"Econ Vault Operations","message":"New request arrived","tran_info":{"deal_id":"1052368321","functional_area":"ECONAFFIX","dealer_code":141348,"user_code":362266,"request_type":"CC"},"extended_properties":{"IP":"10.133.148.159|::1","TranSid":"66909516","ProfSid":"66909518"}}
{"flow_id":"2xxv2neh4f1","correlation_id":"21dcd29f-dc4f-4c6e-8b9e-993c4ce47971","tenant_code":"VAULTTESTTENANT","fusion_prod_code":"VAULTTEST","event_name":"ECSDeposit","event_timestamp":"01/29/2018 10:22:43.622","duration":0.0,"request_url":"http://localsite:61018/va/contact.svc/econtacts","machine_name":"HostD002","method_name":"DT.Service.EcontractVault.HttpListener.EcontractVault.SignbindAndDepositContract","title":"Econ Vault Operations","message":"New request arrived","tran_info":{"deal_id":"828082541","functional_area":"ECONAFFIX","dealer_code":141348,"user_code":362266,"lender_id":"DTL","request_type":"Signing"},"extended_properties":{"IP":"10.133.148.159|::1"}}
{"flow_id":"kiozeybzkmg","correlation_id":"d5589ebc-d17a-4b0e-b7f1-2e754c20e009","tenant_code":"VAULTTESTTENANT","fusion_prod_code":"VAULTTEST","event_name":"ECSDeposit","event_timestamp":"01/29/2018 10:22:44.595","duration":2947.1757000000002,"request_url":"http://localsite:61018/va/contact.svc/econtacts","machine_name":"HostD002","method_name":"","title":"","message":"Request processed successfully","tran_info":{"deal_id":"835514948","functional_area":"ECONAFFIX","dealer_code":141348,"user_code":362266,"lender_id":"DTL","dealer_id":"146387","request_type":"Signing"},"extended_properties":{"IP":"10.133.148.159|::1"}}

This is what I have in my transforms.conf:

 KV_MODE = json
 LINE_BREAKER = ([\r\n]+)(?=\s*\{\s*\"flow_id\")
 #TRUNCATE = 0
 SHOULD_LINEMERGE=false
 NO_BINARY_CHECK=true
 TIME_FORMAT= \"%m/%d/%Y %H:%M:%S.%3N\"
 TIME_PREFIX = \s*\"event_timestamp\"
 MAX_TIMESTAMP_LOOKAHEAD=26
 #BREAK_ONLY_BEFORE = \{\s*\"flow_id\"

Re: Why is the event line breaking not working properly for JSON format?

mayurr98 — Tue, 30 Jan 2018 07:32:43 GMT

hey, I ingested above data in with the following configuration and it worked for me. And you have to write this in props.conf and NOT transforms.conf

[<sourcetype>]
DATETIME_CONFIG =
KV_MODE = json
MAX_TIMESTAMP_LOOKAHEAD = 26
NO_BINARY_CHECK = true
SHOULD_LINEMERGE = false
TIME_FORMAT = %m/%d/%Y %H:%M:%S.%3N
TIME_PREFIX = event_timestamp\":\"
category = Custom
pulldown_type = true

let me know if this helps!

Re: Why is the event line breaking not working properly for JSON format?

apoorvaaj — Tue, 29 Sep 2020 17:55:19 GMT

Hey, @mayurr98 is right, you should save those settings in props.conf and not in transforms.conf

Also, I slightly updated the time properties :

[ sourcetype]
CHARSET=UTF-8
KV_MODE=JSON
NO_BINARY_CHECK=true
SHOULD_LINEMERGE=false
TIME_FORMAT=%m/%d/%Y %H:%M:%S.%3N
TIME_PREFIX="event_timestamp":"
disabled=false
pulldown_type=true

topic Re: Why is the event line breaking not working properly for JSON format? in Getting Data In

Why is the event line breaking not working properly for JSON format?

answer-614348

Re: Why is the event line breaking not working properly for JSON format?

Re: Why is the event line breaking not working properly for JSON format?