Getting Data In

JSON transformations

jackson_storm
Explorer

Hi. I have a problem with transformations in Splunk:

Example event(small part of it):
Dec 1 22:29:42 127.0.0.1 1 2017-12-01 LOGSERVER 1292 - - {"event_type":"type_here","ipv4":"127.0.0.1","hostname":"pc_name.local","occured":"01-Dec-2017 22:24:34"}

I need extract only json data:
{"event_type":"Threat_Event","ipv4":"127.0.0.1","hostname":"pc_name.local","occured":"01-Dec-2017 22:24:34","severity":"Warning","threat_type":"potentially unsafe application"}

and get fields from it.

I'm found one solution:
transforms.conf
[json_extract]
SOURCE_KEY = _raw
DEST_KEY = _raw
REGEX = ^([^{]+)({.+})$
FORMAT = $2

props.conf
[some_sourcetype]
KV_MODE = json
TRANSFORMS-jsonextraction = json_extract

It works! and all ok. But i need to do this using Field Transformations in Splunk Web.

When i'm doing this extractions in Splunk Web, i have problem with Format(more on the screenshot)
Please, help me to do this using SplunkWeb.
alt text

0 Karma
1 Solution

micahkemp
Champion

After some discussion, it is possible to configure SEDCMD via the UI (but not TRANSFORMS- that rewrite _raw). To do so you can edit the sourcetype at:

<your splunk url>/en-US/manager/launcher/sourcetypes#

The select your sourcetype, expand Advanced, and add a line for SEDCMD as previously discussed. Attached is a screenshot of this configuration.

Credit to @duckfez and @martin_mueller

screenshot showing Advanced settings, with SEDCMD

View solution in original post

0 Karma

micahkemp
Champion

After some discussion, it is possible to configure SEDCMD via the UI (but not TRANSFORMS- that rewrite _raw). To do so you can edit the sourcetype at:

<your splunk url>/en-US/manager/launcher/sourcetypes#

The select your sourcetype, expand Advanced, and add a line for SEDCMD as previously discussed. Attached is a screenshot of this configuration.

Credit to @duckfez and @martin_mueller

screenshot showing Advanced settings, with SEDCMD

0 Karma

jackson_storm
Explorer

It works. Thank you

0 Karma

sshelly_splunk
Splunk Employee
Splunk Employee
I would do something like the following:
In your props.conf stanza defining sourcetype:
#This should everything up to and including the first "{" with a "{" - I think
SEDCMD=s/.+?{/{/  

In transforms, something like the following. This will extract key/value pairs at search time for you. Also, make sure to set time_format correctly.
[get_my_json_key_values]
REGEX="(.+?)"."(.+?)"
FORMAT=$1::$2
0 Karma

micahkemp
Champion

I think the issue is finding a way to accomplish this in splunkweb, instead of directly in .conf files.

sshelly_splunk
Splunk Employee
Splunk Employee
        Sorry - a bit quick on the "submit" button:)
        In the UI, you can create the sourcetype. I used your single event, and this is what I came up with: 
    Timestamp format %d-%b-%Y %H:%M:%S
    Timestamp prefix occured":"
    Under "Advanced" 
    SEDCMD s/.+?{/{/
    KV_MODE=json
0 Karma

sshelly_splunk
Splunk Employee
Splunk Employee
    Sorry - a bit quick on the "submit" button:)
    In the UI, you can create the sourcetype. I used your single event, and this is what I came up with: 
Timestamp format %d-%b-%Y %H:%M:%S
Timestamp prefix occured":"
Under "Advanced" 
SEDCMD s/.+?{/{/
KV_MODE=json
0 Karma

jackson_storm
Explorer

Yes. The issue is finding a way to accomplish this in splunkweb, instead of directly in .conf files.

0 Karma

micahkemp
Champion

Are you having to use the UI to do this because it's Splunk Cloud?

I've been unable to get this working without setting DEST_KEY in transforms.conf, which isn't possible via the UI that I can tell, so you may not actually be able to accomplish this in Splunk Cloud (though I'm going to keep trying, or maybe someone else already knows how).

If it's not Splunk Cloud, you may need to send your transforms/props to the admin to put in place. Or, you could put said configurations in place via conf file on a heavy forwarder you maintain which sends the events to the indexers.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...