topic Re: Can I create a field transformation using a JSON source key? in Getting Data In

Can I create a field transformation using a JSON source key?

Jordan_Brough — Mon, 28 Sep 2020 14:12:42 GMT

I don't seem to be able to set up a field transformation using a Source Key that comes from a JSON event field.

I have events like this:

{
  "time": "2013-06-23T13:55:37+00:00",
  "handler": "UsersController#index"
}

And I'd like to extract "UsersController" and "index" from the "handler" field. I have props.conf configured with KV_MODE=json

I added this transform via the GUI:

[controller_action_transform]
CLEAN_KEYS = 1
MV_ADD = 0
REGEX = (?<controller>.*)#(?<action>.*)
SOURCE_KEY = handler

and this field extraction:

[json]
REPORT-controller_action_extraction = controller_action_transform

however, when I do a query like this:

sourcetype=json | table handler controller action

I do get results for "handler" but don't get anything for "controller" or "action":

| handler               | controller | action |
-----------------------------------------------
| UsersController#index |            |        |

If I change the transform SOURCE_KEY to "_raw" then I do get results for controller & action (though not exactly correct).

Also, I can do an inline "rex" field extraction using the "handler" field and get the correct results. That is, this works just fine:

sourcetype=json | rex field=handler "(?<controller>.*)#(?<action>.*)" | table handler controller action

Am I doing something wrong with the transform? Are JSON-extracted fields not available for use in transforms or something?

(NOTE: The above is just some sample data I created for testing this out. The real logs that I need to use this on have more data and nested keys and so forth, so a workaround that involves not using the extracted JSON fields would be pretty non-ideal.)

Re: Can I create a field transformation using a JSON source key?

Jordan_Brough — Mon, 28 Sep 2020 14:47:01 GMT

I got an answer via Splunk Support.

They said:

I was able to repro. And this is a bug.

If not using the SOURCE_KEY then extractions are working.

When referring a field that was extracted using KV_MODE = json, it is not working.

I was able to find an already existing issue for that bug.

It's SPL-61046 and will probably fix in the next major release. (not maintenance release)

But there is a workaround to get it working.

Use the search like:

sourcetype=json | kv reload=t | table handler controller action

I've confirmed that the workaround does solve the problem.

Re: Can I create a field transformation using a JSON source key?

apringle — Tue, 29 Sep 2020 06:59:45 GMT

I was just curious if the referenced SPL-61046 issue was ever resolved? I am trying to do something very similar to the OP and having the same issue. I'm able to get around it by using a regex match on the _raw data, but it would be nice to be able to define the SOURCE_KEY for the JSON data.

(Also, if there is somewhere that I can view details about the referenced SPL issue, please let me know)

Re: Can I create a field transformation using a JSON source key?

phoenixdigital — Wed, 17 Aug 2016 03:48:06 GMT

3 years on I am still seeing this issue.

Does anyone know at what point JSON fields are extracted?

It appears to be after custom transforms.conf configs.

Re: Can I create a field transformation using a JSON source key?

regiteric — Thu, 05 Jan 2017 00:46:15 GMT

Got the same issue, doing the transformation on the _raw seems to be the only solution. But is is not perfect as the escaped character in the JSON value are not displayed correctly.

Re: Can I create a field transformation using a JSON source key?

vliggio — Tue, 29 Sep 2020 12:20:38 GMT

I ping'ed my splunk support rep today and was told:

"The bug was closed with "cannot reproduce" this past October. The original issue was reported for 4.1.3 and 5.0.1.

That said, Splunk is particular about the JSON. Extraneous and/or incorrect delimiters will cause extraction to fail. Additionally, you should set KV_MODE=none if you are using INDEXED_EXTRACTIONS=json, otherwise data will be duplicated."

Re: Can I create a field transformation using a JSON source key?

mikaelbje — Mon, 25 Jun 2018 08:17:19 GMT

I'm also trying to get this working. The only way I got it working was using the | kv reload=t trick. I tried both with KV_MODE = json and with the default. Data is ingested using the HTTP Event Collector.

Splunk 7.1.0. SPL-61046 should be reopened IMHO

Re: Can I create a field transformation using a JSON source key?

mikaelbje — Mon, 25 Jun 2018 08:23:30 GMT

I was wrong. KV_MODE = json solved it. No need for the| kv reload=t trick