Getting Data In

JSON transformations

Explorer

Hi. I have a problem with transformations in Splunk:

Example event(small part of it):
Dec 1 22:29:42 127.0.0.1 1 2017-12-01 LOGSERVER 1292 - - {"event_type":"type_here","ipv4":"127.0.0.1","hostname":"pc_name.local","occured":"01-Dec-2017 22:24:34"}

I need extract only json data:
{"event_type":"Threat_Event","ipv4":"127.0.0.1","hostname":"pc_name.local","occured":"01-Dec-2017 22:24:34","severity":"Warning","threat_type":"potentially unsafe application"}

and get fields from it.

I'm found one solution:
transforms.conf
[json_extract]
SOURCE_KEY = _raw
DEST_KEY = _raw
REGEX = ^([^{]+)({.+})$
FORMAT = $2

props.conf
[some_sourcetype]
KV_MODE = json
TRANSFORMS-jsonextraction = json_extract

It works! and all ok. But i need to do this using Field Transformations in Splunk Web.

When i'm doing this extractions in Splunk Web, i have problem with Format(more on the screenshot)
Please, help me to do this using SplunkWeb.
alt text

0 Karma
1 Solution

Champion

After some discussion, it is possible to configure SEDCMD via the UI (but not TRANSFORMS- that rewrite _raw). To do so you can edit the sourcetype at:

<your splunk url>/en-US/manager/launcher/sourcetypes#

The select your sourcetype, expand Advanced, and add a line for SEDCMD as previously discussed. Attached is a screenshot of this configuration.

Credit to @duckfez and @martin_mueller

screenshot showing Advanced settings, with SEDCMD

View solution in original post

0 Karma

Champion

After some discussion, it is possible to configure SEDCMD via the UI (but not TRANSFORMS- that rewrite _raw). To do so you can edit the sourcetype at:

<your splunk url>/en-US/manager/launcher/sourcetypes#

The select your sourcetype, expand Advanced, and add a line for SEDCMD as previously discussed. Attached is a screenshot of this configuration.

Credit to @duckfez and @martin_mueller

screenshot showing Advanced settings, with SEDCMD

View solution in original post

0 Karma

Explorer

It works. Thank you

0 Karma

Splunk Employee
Splunk Employee
I would do something like the following:
In your props.conf stanza defining sourcetype:
#This should everything up to and including the first "{" with a "{" - I think
SEDCMD=s/.+?{/{/  

In transforms, something like the following. This will extract key/value pairs at search time for you. Also, make sure to set time_format correctly.
[get_my_json_key_values]
REGEX="(.+?)"."(.+?)"
FORMAT=$1::$2
0 Karma

Champion

I think the issue is finding a way to accomplish this in splunkweb, instead of directly in .conf files.

Splunk Employee
Splunk Employee
        Sorry - a bit quick on the "submit" button:)
        In the UI, you can create the sourcetype. I used your single event, and this is what I came up with: 
    Timestamp format %d-%b-%Y %H:%M:%S
    Timestamp prefix occured":"
    Under "Advanced" 
    SEDCMD s/.+?{/{/
    KV_MODE=json
0 Karma

Splunk Employee
Splunk Employee
    Sorry - a bit quick on the "submit" button:)
    In the UI, you can create the sourcetype. I used your single event, and this is what I came up with: 
Timestamp format %d-%b-%Y %H:%M:%S
Timestamp prefix occured":"
Under "Advanced" 
SEDCMD s/.+?{/{/
KV_MODE=json
0 Karma

Explorer

Yes. The issue is finding a way to accomplish this in splunkweb, instead of directly in .conf files.

0 Karma

Champion

Are you having to use the UI to do this because it's Splunk Cloud?

I've been unable to get this working without setting DEST_KEY in transforms.conf, which isn't possible via the UI that I can tell, so you may not actually be able to accomplish this in Splunk Cloud (though I'm going to keep trying, or maybe someone else already knows how).

If it's not Splunk Cloud, you may need to send your transforms/props to the admin to put in place. Or, you could put said configurations in place via conf file on a heavy forwarder you maintain which sends the events to the indexers.

0 Karma
State of Splunk Careers

Access the Splunk Careers Report to see real data that shows how Splunk mastery increases your value and job satisfaction.

Find out what your skills are worth!