topic Re: How can we extract a json document within an event? in Getting Data In

How can we extract a json document within an event?

ddrillic — Tue, 10 Oct 2017 14:51:15 GMT

We have events such as -

10.10.2017 09:40:39.651 *INFO* [10.86.208.119 [1507646439651] POST /apps/xxxx/yyyy HTTP/1.1] com.xxxx.yyyy.api.impl.logging.info.InfoLoggerServiceImpl {"id":{"access_token":"7ee2ea18-e72c-449d-9dec-28d02b116c92","uid":"zzzzz","jsessionID":"aaaaaaa","uuid":"12e255ac-35e9-4630-a36b-89aa27e9566e"},"request":{"url":"https://bbb.cccc.com/content/uuuuuu"..... }]}}

The json document is part of the event. Can we extract this json document?

Re: How can we extract a json document within an event?

sshelly_splunk — Tue, 29 Sep 2020 16:13:12 GMT

I took a quick look at this, and I think this transforms might work for you. This will not get the "id" or "request" fields, as I am not sure what they are. This did get the following: access_token, uid, jsessionID, uuid and url.
In props, I added: REPORT-extract = json_embedded
The transforms stanza is:
[json_embedded]
REGEX = "(\w+)"."(\S+?)"
FORMAT = $1::$2

Hope this helps. Reply if it does not.

Re: How can we extract a json document within an event?

ddrillic — Tue, 10 Oct 2017 15:46:48 GMT

Beautiful thing!!! I wanted to ask for a while - is there a way to test these configurations somehow from the search interface before adding these configurations to the config files?

Re: How can we extract a json document within an event?

sbbadri — Tue, 10 Oct 2017 16:06:00 GMT

https://regex101.com/r/FPxKuU/1

| makeresults | eval test="10.10.2017 09:40:39.651 INFO [10.86.208.119 [1507646439651] POST /apps/xxxx/yyyy HTTP/1.1] com.xxxx.yyyy.api.impl.logging.info.InfoLoggerServiceImpl {\"id\":{\"access_token\":\"7ee2ea18-e72c-449d-9dec-28d02b116c92\",\"uid\":\"zzzzz\",\"jsessionID\":\"aaaaaaa\",\"uuid\":\"12e255ac-35e9-4630-a36b-89aa27e9566e\"},\"request\":{\"url\":\"https://bbb.cccc.com/content/uuuuuu\"..... }]}}" | rex field=test "(?P\"(\w+)\".\"(\S+))\""

Re: How can we extract a json document within an event?

ddrillic — Tue, 10 Oct 2017 16:08:44 GMT

Wow - man. very pretty!!!

Re: How can we extract a json document within an event?

sshelly_splunk — Tue, 10 Oct 2017 16:14:47 GMT

I use regex101 to test all of my transforms (unless they are extremely simple:)).
Copy 2 events if available into the "Test String" window, and go to town.

Re: How can we extract a json document within an event?

sshelly_splunk — Tue, 10 Oct 2017 16:16:11 GMT

sorry - just re-read your question. I test regex in the search bar sometimes, but not usually. Slightly different format, etc, so I use regex101, but might be just a preference.

Re: How can we extract a json document within an event?

ddrillic — Tue, 10 Oct 2017 16:19:41 GMT

Perfect, so I got the REGEX part. What does the FORMAT - $1::$2 mean?

Re: How can we extract a json document within an event?

blacknight659 — Tue, 10 Oct 2017 17:10:35 GMT

You could take a raw copy of the logs and use the UI to upload and test the event breaking and extraction. I think Splunk really likes Json since it auto extracts the fields and values.

Re: How can we extract a json document within an event?

ddrillic — Wed, 11 Oct 2017 22:03:36 GMT

Just applied it and it works perfectly - much appreciated. Just wondering if there is anything like the spath command that we use for XML documents for json documents, so we can reach nested elements ...

Re: How can we extract a json document within an event?

sshelly_splunk — Thu, 12 Oct 2017 14:08:20 GMT

ddrillic - You can index just the json portion of the event, but it looks like the text before the json portion includes timestamp, etc. Since this log is not proper json, I think you're going to need to do regex on it for display purposes.

When looking at xml or json data (assuming it conforms to standards - sorry not exactly sure what that all entails:)), you can use kvmode=xml or json, or use something like the above. My skills are really around getting data in, and not SPL proper (I know, I know:)), so I will defer to the SPL experts for the spl-specific questions, but my focus is really on making sure data comes in correctly, so the SPL doesn't need to be complex to get value out of the data. Sorry if that doesnt help.

Re: How can we extract a json document within an event?

ddrillic — Fri, 13 Oct 2017 15:23:09 GMT

Very interesting, so you are saying that if it's a "real" json document we can parse it as such - interesting.

Re: How can we extract a json document within an event?

ddrillic — Mon, 16 Oct 2017 23:11:33 GMT

For future reference -

FORMAT = $1::$2 (where the REGEX extracts both the field name and the field value)

from Create custom fields at index time