topic Re: JSON transformations in Getting Data In

JSON transformations

jackson_storm — Tue, 02 Jan 2018 21:05:29 GMT

Hi. I have a problem with transformations in Splunk:

Example event(small part of it):
Dec 1 22:29:42 127.0.0.1 1 2017-12-01 LOGSERVER 1292 - - {"event_type":"type_here","ipv4":"127.0.0.1","hostname":"pc_name.local","occured":"01-Dec-2017 22:24:34"}

I need extract only json data:
{"event_type":"Threat_Event","ipv4":"127.0.0.1","hostname":"pc_name.local","occured":"01-Dec-2017 22:24:34","severity":"Warning","threat_type":"potentially unsafe application"}

and get fields from it.

I'm found one solution:
transforms.conf
[json_extract] SOURCE_KEY = _raw DEST_KEY = _raw REGEX = ^([^{]+)({.+})$ FORMAT = $2

props.conf
[some_sourcetype] KV_MODE = json TRANSFORMS-jsonextraction = json_extract

It works! and all ok. But i need to do this using Field Transformations in Splunk Web.

When i'm doing this extractions in Splunk Web, i have problem with Format(more on the screenshot)
Please, help me to do this using SplunkWeb.

Re: JSON transformations

micahkemp — Tue, 02 Jan 2018 23:34:22 GMT

Are you having to use the UI to do this because it's Splunk Cloud?

I've been unable to get this working without setting DEST_KEY in transforms.conf, which isn't possible via the UI that I can tell, so you may not actually be able to accomplish this in Splunk Cloud (though I'm going to keep trying, or maybe someone else already knows how).

If it's not Splunk Cloud, you may need to send your transforms/props to the admin to put in place. Or, you could put said configurations in place via conf file on a heavy forwarder you maintain which sends the events to the indexers.

Re: JSON transformations

sshelly_splunk — Wed, 03 Jan 2018 00:02:27 GMT

I would do something like the following:
In your props.conf stanza defining sourcetype:
#This should everything up to and including the first "{" with a "{" - I think
SEDCMD=s/.+?{/{/  

In transforms, something like the following. This will extract key/value pairs at search time for you. Also, make sure to set time_format correctly.
[get_my_json_key_values]
REGEX="(.+?)"."(.+?)"
FORMAT=$1::$2

Re: JSON transformations

micahkemp — Wed, 03 Jan 2018 00:05:31 GMT

I think the issue is finding a way to accomplish this in splunkweb, instead of directly in .conf files.

Re: JSON transformations

jackson_storm — Wed, 03 Jan 2018 10:06:37 GMT

Yes. The issue is finding a way to accomplish this in splunkweb, instead of directly in .conf files.

Re: JSON transformations

sshelly_splunk — Wed, 03 Jan 2018 16:17:11 GMT

    Sorry - a bit quick on the "submit" button:)
    In the UI, you can create the sourcetype. I used your single event, and this is what I came up with: 
Timestamp format %d-%b-%Y %H:%M:%S
Timestamp prefix occured":"
Under "Advanced" 
SEDCMD s/.+?{/{/
KV_MODE=json

Re: JSON transformations

sshelly_splunk — Wed, 03 Jan 2018 16:18:01 GMT

        Sorry - a bit quick on the "submit" button:)
        In the UI, you can create the sourcetype. I used your single event, and this is what I came up with: 
    Timestamp format %d-%b-%Y %H:%M:%S
    Timestamp prefix occured":"
    Under "Advanced" 
    SEDCMD s/.+?{/{/
    KV_MODE=json

Re: JSON transformations

micahkemp — Wed, 03 Jan 2018 16:42:30 GMT

After some discussion, it is possible to configure SEDCMD via the UI (but not TRANSFORMS- that rewrite _raw). To do so you can edit the sourcetype at:

<your splunk url>/en-US/manager/launcher/sourcetypes#

The select your sourcetype, expand Advanced, and add a line for SEDCMD as previously discussed. Attached is a screenshot of this configuration.

Credit to @duckfez and @martin_mueller

Re: JSON transformations

jackson_storm — Thu, 04 Jan 2018 10:29:36 GMT

It works. Thank you