topic Re: how to forward JSON data to splunk Properly in Getting Data In

how to forward JSON data to splunk Properly

kirrusk — Wed, 25 Nov 2020 13:32:03 GMT

I have a json file like below, i need to broke it up in to events

{"env":"UAT","label":"jenkins-17887.api.v2.dm.btc","App":"dm-d-services","rlmtemplate":"f2_api_fed","lastupdate":2020-11-23 11:09:78:455,"region":"APAC"},{"env":"UAT","label":"jenkins-17687.api.v2.dm.btc","App":"dt-s-services","rlmtemplate":"f3_api_fed","lastupdate":2020-11-23 11:025:79:475,"region":"APAC"},{"env":"UAT","label":"jenkins-18657.api.v2.dm.btc","App":"dt-s-services","rlmtemplate":"f3_api_fed","lastupdate":2020-11-23 11:025:79:475,"region":"APAC"},{"env":"UAT","label":"jenkins-17637.api.v2.dm.btc","App":"dt-s-services","rlmtemplate":"f3_api_fed","lastupdate":2020-11-23 11:025:79:475,"region":"APAC"}

I'm trying to forward it to splunk

modified props.conf file like below

[test_json]

INDEXED_EXTRACTIONS = JSON

LINEBREAKER = }(,){"env":

SHOULD_LINEMERGE = false

NO_BINARY_CHECK = true

TRUNCATE = 0

TZ = Asia/Singapore

But only getting first line of json as event , remaining data is not coming to splunk.

==Firstline ==

"env":"UAT","label":"jenkins-17887.api.v2.dm.btc","App":"dm-d-services","rlmtemplate":"f2_api_fed","lastupdate":2020-11-23 11:09:78:455,"region":"APAC"

Can any one suggest what's going wrong.

Re: how to forward JSON data to splunk Properly

to4kawa — Wed, 25 Nov 2020 13:56:03 GMT

[test_json] DATETIME_CONFIG = KV_MODE = json LINE_BREAKER = (,){ NO_BINARY_CHECK = true SHOULD_LINEMERGE = false TIME_FORMAT = %F %H:%M TIME_PREFIX = lastupdate\":

indexed_extractions can't work.

Re: how to forward JSON data to splunk Properly

richgalloway — Wed, 25 Nov 2020 14:15:41 GMT

"LINEBREAKER" should be "LINE_BREAKER". The props must be installed on the first heavy forwarder or indexer the events pass through. Don't forget to restart Splunk after modifying props.conf.

Re: how to forward JSON data to splunk Properly

kirrusk — Wed, 25 Nov 2020 14:48:13 GMT

@richgalloway I have used LINE_BREAKER , and props is installed on indexer. Still not working.

Re: how to forward JSON data to splunk Properly

kirrusk — Wed, 25 Nov 2020 14:49:27 GMT

@to4kawa Modified props but it's giving single event without breaking json

Re: how to forward JSON data to splunk Properly

burwell — Wed, 25 Nov 2020 20:05:43 GMT

Hi. What simplified things for me was to ask the people creating logs not to have the trailing comma. So the logs look like this

{"timestamp": "2020-11-25 08:59:24 UTC", "hostname": "foo.com", "status": "failed"}
{"timestamp": "2020-11-25 08:59:29 UTC", "hostname": "bar.com", "status": "passed"}

Props are

[my_sourcetype] INDEXED_EXTRACTIONS = JSON KV_MODE = none SHOULD_LINEMERGE=false TIME_PREFIX = timestamp\":\s*\" TIME_FORMAT=%Y-%m-%d %H:%M:%S %Z MAX_TIMESTAMP_LOOKAHEAD=23 TRUNCATE=99999

Re: how to forward JSON data to splunk Properly

kirrusk — Thu, 26 Nov 2020 07:16:59 GMT

Thank you all, it's working for me after removing comma's using sed command.