JSON transformations

[jackson_storm]

Hi. I have a problem with transformations in Splunk:

Example event(small part of it):
Dec 1 22:29:42 127.0.0.1 1 2017-12-01 LOGSERVER 1292 - - {"event_type":"type_here","ipv4":"127.0.0.1","hostname":"pc_name.local","occured":"01-Dec-2017 22:24:34"}

I need extract only json data:
{"event_type":"Threat_Event","ipv4":"127.0.0.1","hostname":"pc_name.local","occured":"01-Dec-2017 22:24:34","severity":"Warning","threat_type":"potentially unsafe application"}

and get fields from it.

I'm found one solution:
transforms.conf
[json_extract]
SOURCE_KEY = _raw
DEST_KEY = _raw
REGEX = ^([^{]+)({.+})$
FORMAT = $2

props.conf
[some_sourcetype]
KV_MODE = json
TRANSFORMS-jsonextraction = json_extract

It works! and all ok. But i need to do this using Field Transformations in Splunk Web.

When i'm doing this extractions in Splunk Web, i have problem with Format(more on the screenshot)
Please, help me to do this using SplunkWeb.

[micahkemp · Accepted solution]

After some discussion, it is possible to configure SEDCMD via the UI (but not TRANSFORMS- that rewrite _raw). To do so you can edit the sourcetype at:

<your splunk url>/en-US/manager/launcher/sourcetypes#

The select your sourcetype, expand Advanced, and add a line for SEDCMD as previously discussed. Attached is a screenshot of this configuration.

Credit to @duckfez and @martin_mueller

[jackson_storm]

It works. Thank you

[sshelly_splunk]

I would do something like the following:
In your props.conf stanza defining sourcetype:
#This should everything up to and including the first "{" with a "{" - I think
SEDCMD=s/.+?{/{/  

In transforms, something like the following. This will extract key/value pairs at search time for you. Also, make sure to set time_format correctly.
[get_my_json_key_values]
REGEX="(.+?)"."(.+?)"
FORMAT=$1::$2

[micahkemp]

I think the issue is finding a way to accomplish this in splunkweb, instead of directly in .conf files.

[sshelly_splunk]

        Sorry - a bit quick on the "submit" button:)
        In the UI, you can create the sourcetype. I used your single event, and this is what I came up with: 
    Timestamp format %d-%b-%Y %H:%M:%S
    Timestamp prefix occured":"
    Under "Advanced" 
    SEDCMD s/.+?{/{/
    KV_MODE=json

[sshelly_splunk]

    Sorry - a bit quick on the "submit" button:)
    In the UI, you can create the sourcetype. I used your single event, and this is what I came up with: 
Timestamp format %d-%b-%Y %H:%M:%S
Timestamp prefix occured":"
Under "Advanced" 
SEDCMD s/.+?{/{/
KV_MODE=json

[jackson_storm]

Yes. The issue is finding a way to accomplish this in splunkweb, instead of directly in .conf files.

[micahkemp]

Are you having to use the UI to do this because it's Splunk Cloud?

I've been unable to get this working without setting DEST_KEY in transforms.conf, which isn't possible via the UI that I can tell, so you may not actually be able to accomplish this in Splunk Cloud (though I'm going to keep trying, or maybe someone else already knows how).

If it's not Splunk Cloud, you may need to send your transforms/props to the admin to put in place. Or, you could put said configurations in place via conf file on a heavy forwarder you maintain which sends the events to the indexers.