Getting Data In

how to forward JSON data to splunk Properly

kirrusk
Communicator

I have a json file like below, i need to broke it up in to events

{"env":"UAT","label":"jenkins-17887.api.v2.dm.btc","App":"dm-d-services","rlmtemplate":"f2_api_fed","lastupdate":2020-11-23 11:09:78:455,"region":"APAC"},{"env":"UAT","label":"jenkins-17687.api.v2.dm.btc","App":"dt-s-services","rlmtemplate":"f3_api_fed","lastupdate":2020-11-23 11:025:79:475,"region":"APAC"},{"env":"UAT","label":"jenkins-18657.api.v2.dm.btc","App":"dt-s-services","rlmtemplate":"f3_api_fed","lastupdate":2020-11-23 11:025:79:475,"region":"APAC"},{"env":"UAT","label":"jenkins-17637.api.v2.dm.btc","App":"dt-s-services","rlmtemplate":"f3_api_fed","lastupdate":2020-11-23 11:025:79:475,"region":"APAC"}

I'm trying to forward it to splunk

modified props.conf file like below

[test_json]

INDEXED_EXTRACTIONS = JSON

LINEBREAKER = }(,){"env":

SHOULD_LINEMERGE = false

NO_BINARY_CHECK = true

TRUNCATE = 0

TZ = Asia/Singapore

 

But only getting first line of json as event , remaining data is not coming to splunk.

==Firstline ==

"env":"UAT","label":"jenkins-17887.api.v2.dm.btc","App":"dm-d-services","rlmtemplate":"f2_api_fed","lastupdate":2020-11-23 11:09:78:455,"region":"APAC"

Can any one suggest what's going wrong.

 

 

 

0 Karma
1 Solution

kirrusk
Communicator

Thank you all,  it's working for me after removing comma's using sed command.

View solution in original post

0 Karma

kirrusk
Communicator

Thank you all,  it's working for me after removing comma's using sed command.

0 Karma

burwell
SplunkTrust
SplunkTrust

Hi.  What simplified things for me was to ask the people creating logs not to have the trailing comma. So the logs look like this

{"timestamp": "2020-11-25 08:59:24 UTC", "hostname": "foo.com", "status": "failed"}
{"timestamp": "2020-11-25 08:59:29 UTC", "hostname": "bar.com", "status": "passed"}

Props are

[my_sourcetype]
INDEXED_EXTRACTIONS = JSON
KV_MODE = none
SHOULD_LINEMERGE=false
TIME_PREFIX = timestamp\":\s*\"
TIME_FORMAT=%Y-%m-%d %H:%M:%S %Z
MAX_TIMESTAMP_LOOKAHEAD=23
TRUNCATE=99999

 

0 Karma

richgalloway
SplunkTrust
SplunkTrust

"LINEBREAKER" should be "LINE_BREAKER".  The props must be installed on the first heavy forwarder or indexer the events pass through.  Don't forget to restart Splunk after modifying props.conf.  

---
If this reply helps you, Karma would be appreciated.
0 Karma

kirrusk
Communicator

@richgalloway  I have used LINE_BREAKER , and props is installed on indexer. Still not working.

0 Karma

to4kawa
Ultra Champion
[test_json]
DATETIME_CONFIG = 
KV_MODE = json
LINE_BREAKER = (,){
NO_BINARY_CHECK = true
SHOULD_LINEMERGE = false
TIME_FORMAT = %F %H:%M
TIME_PREFIX = lastupdate\":

indexed_extractions can't work.

0 Karma

kirrusk
Communicator

@to4kawa Modified props but it's giving single event without breaking json

0 Karma
Get Updates on the Splunk Community!

Built-in Service Level Objectives Management to Bridge the Gap Between Service & ...

Wednesday, May 29, 2024  |  11AM PST / 2PM ESTRegister now and join us to learn more about how you can ...

Get Your Exclusive Splunk Certified Cybersecurity Defense Engineer Certification at ...

We’re excited to announce a new Splunk certification exam being released at .conf24! If you’re headed to Vegas ...

Share Your Ideas & Meet the Lantern team at .Conf! Plus All of This Month’s New ...

Splunk Lantern is Splunk’s customer success center that provides advice from Splunk experts on valuable data ...