topic Re: How do we assign each JSON document to a distinct event? in Getting Data In

How do we assign each JSON document to a distinct event?

ddrillic — Thu, 21 Dec 2017 22:48:04 GMT

We have a case in which multiple json documents are being clamped together into one Splunk event. How do we untangle it?

Re: How do we assign each JSON document to a distinct event?

harsmarvania57 — Fri, 22 Dec 2017 02:43:26 GMT

Hi @ddrillic,

Can you please provide some sample data?

Re: How do we assign each JSON document to a distinct event?

niketn — Fri, 22 Dec 2017 03:17:34 GMT

@ddrillic also add what is your current sourcetype stanza for JSON data?

Re: How do we assign each JSON document to a distinct event?

DavidHourani — Sat, 23 Dec 2017 11:48:10 GMT

Hi ddrillic,

This usually happens when you have brackets at the beginning of your JSON containing the entire document. It makes it as if the entire document is a value for one of the elements. You should set up a sedcmd in your props to clear this up, or clear it via script before the data gets into Splunk.

If you post a copy of the header/end of your JSON file I can help you set up the sedcmd.

Regards,
David

Re: How do we assign each JSON document to a distinct event?

ddrillic — Wed, 03 Jan 2018 22:06:59 GMT

@niketnilay, sorry for the delay. We didn't set anything in the configuration files.

Re: How do we assign each JSON document to a distinct event?

ddrillic — Wed, 03 Jan 2018 22:12:33 GMT

Interesting - it looks like {"userDetails":{...."message":null} followed by another one like this one - {"userDetails":{...."message":null}...

Re: How do we assign each JSON document to a distinct event?

somesoni2 — Wed, 03 Jan 2018 22:14:51 GMT

You would need to set appropriate Line breaking configuration for your sourcetype, and for which we'd need some sample data (mask anything that's sensitive), and some details on how you'd want to break that sample event.

Re: How do we assign each JSON document to a distinct event?

ddrillic — Wed, 03 Jan 2018 22:29:26 GMT

It looks like -

{"userDetails":{sensitive data},"message":null}
{"userDetails":{sensitive data},"message":null}
{"userDetails":{sensitive data},"message":null}
{"userDetails":{sensitive data},"message":null}

Re: How do we assign each JSON document to a distinct event?

somesoni2 — Wed, 03 Jan 2018 22:40:21 GMT

Try to use following in props.conf on Indexer(s)/Heavy Forwarder(s) whichever comes first.

[YourSourceTypeHere]
SHOULD_LINEMERGE = false
LINE_BREAKER = ([\r\n]+)(?=\{\"userDetails\"\:)
..other timestamp extraction attributes...

Re: How do we assign each JSON document to a distinct event?

ddrillic — Wed, 03 Jan 2018 22:51:08 GMT

Gorgeous as usual ; -)
But, any way to avoid the hard-coding of userDetails?

Needless to say - working as expected !!!!!!!!!!

Re: How do we assign each JSON document to a distinct event?

somesoni2 — Wed, 03 Jan 2018 22:53:42 GMT

Well, you generally need to put an anchor for identifying line start. You can try with ([\r\n]+)(?=\{\"\w+\"\:) to see if that works for. Since we don't have full events, we can't say for sure that it'll work (there may be other entries matching that pattern).

Re: How do we assign each JSON document to a distinct event?

DavidHourani — Thu, 04 Jan 2018 09:32:33 GMT

if your lines are always starting with a new element you can go for this config :

[yourSourcetype]
BREAK_ONLY_BEFORE = ^\{

Re: How do we assign each JSON document to a distinct event?

skoelpin — Tue, 30 Jan 2018 15:24:57 GMT

LINE_BREAKER would be a much better approach than BREAK_ONLY_BEFORE

Re: How do we assign each JSON document to a distinct event?

DavidHourani — Tue, 30 Jan 2018 17:36:36 GMT

why do you say that ?

Re: How do we assign each JSON document to a distinct event?

skoelpin — Tue, 30 Jan 2018 18:03:14 GMT

If you set SHOULD_LINEMERGE = false and use LINE_BREAKER, this will skip the merging pipeline and give a performance boost

http://wiki.splunk.com/Community:HowIndexingWorks