topic Re: How to use spath command in props.conf or transforms.conf? in Splunk Search

How to use spath command in props.conf or transforms.conf?

leeyounsoo — Wed, 18 Apr 2018 08:57:35 GMT

hello
I have a question

I have a data that access_log data and json data is mixed and my need is to extract field from mixed data.
splunk auto extraction field from that key=value format data
So, i received extracted json field(field name is "json") like this :

{"key":"value", "key":"value", "key":"value", "key":"value", "key":"value" ....(skip)}

And if I search the json field, I return the data but, I want extracted field from JSON like this :

| fields key, key2, key3, key4 ....(skip)

Finally, I found spath command and I got the results that I wanted.
I tried to modify props.conf to automatically extract the field from json but it is not working.

What should I do?

ps: I tried t0 use the eval command like this :

| eval example = spath(json, 'json.key')

And like this:

| eval example = json.key

But that is also not working.

Re: How to use spath command in props.conf or transforms.conf?

ssadanala1 — Tue, 29 Sep 2020 19:03:48 GMT

Please specify

INDEXED_EXTRACTIONS = JSON OR KV_MODE = json but not both .

The above should extract fields at index time for structured json data .

Re: How to use spath command in props.conf or transforms.conf?

niketn — Tue, 29 Sep 2020 19:03:59 GMT

Refer to documentation for KV_MODE : https://docs.splunk.com/Documentation/Splunk/latest/Knowledge/Automatickey-valuefieldextractionsatsearch-time

and INDEXED_EXTRACTIONS: http://docs.splunk.com/Documentation/Splunk/latest/Data/Extractfieldsfromfileswithstructureddata#Props.conf_attributes_for_structured_data

Re: How to use spath command in props.conf or transforms.conf?

leeyounsoo — Fri, 20 Apr 2018 07:34:24 GMT

Sorry.

I think you gave me a slightly different answer because I did not give you the exact details.

The access_log and json data are mixed as follows:
"POST /sample.logs HTTP / 1.1" 200 0 "app: /data.swf ..." "another logs ...." 10.10.10.10 - - [November / 2018: 16: 31: "userAgent data. ... "" json = {"key": "value", "key": "value"

The json field is extracted by the automatic extraction setting of the splunk.
What I want is the value of the json field.

The search command was able to extract the fields with spath command like this :
.... | spath input = json.

How can I get the same results in props.conf or transforms.conf?

thank you for answer

Re: How to use spath command in props.conf or transforms.conf?

woodcock — Sun, 22 Apr 2018 21:16:57 GMT

Go to Settings -> Fields -> Field extractoins -> New.
Enter anything that you like for Name (I suggest something like ColonCommaKVPs), Enter the exact name of your sourcetype in the named field, keep the default of Inline for Type and Sourcetype for Apply to, then enter this for Extraction/Transform:

"(?<_KEY_1>[^"]*)":"(?<_VAL_1>[^"]*)"

Re: How to use spath command in props.conf or transforms.conf?

leeyounsoo — Tue, 29 Sep 2020 19:11:36 GMT

thank you for answer.
but, that solution is not working.
I think there are two reasons.

my data is some problem like this : 1-1) some data has zero string
"json= {"key1":"value1", "key2":"","key3":"value3" .....}"

1-2) some data format is not "key":"value"
- value has blank before "value"
- "json= {"key1": "value1", "key2": "","key3": "value3" .....}"
- "key":(this place is blank)"value"

props.conf overlap
i write props.conf in $SPLUNK_HOME$/etc/system/local/props.conf
your solution place in $SPLUNK_HOME$/etc/apps/$my_app$/local/props.conf
I know that the .conf file under the system folder has a higher priority. So I moved EXTRACT-my = "(? <_ KEY_1> [^"] + ")" (? [^ "] +") to props.conf under the system folder. But it did not work.

What's wrong?

Re: How to use spath command in props.conf or transforms.conf?

woodcock — Mon, 23 Apr 2018 19:36:26 GMT

You should do it exactly as i described (from the GUI) and make it work, then repackage it in whatever app. My solution, with a minor adjustment of changing the + characters to * will work for null values. I have modified my answer so it now reads like this:

"(?<_KEY_1>[^"]*)":"(?<_VAL_1>[^"]*)"

Re: How to use spath command in props.conf or transforms.conf?

leeyounsoo — Tue, 24 Apr 2018 00:00:10 GMT

OK i will tried that.

thank you for answer!!

Re: How to use spath command in props.conf or transforms.conf?

leeyounsoo — Tue, 29 Sep 2020 19:12:38 GMT

hello woodcock.
i tried that and that look like work, but that is static.(my think...
Perhaps, You gave me the answer that I wanted. but
Maybe it is that i did not understand)

so, In my opinion, your answer is need loop like this :
[report-stanza]
"(?<_KEY_1>[^"])":"(?<_VAL_1>[^"])"
"(?<_KEY_2>[^"])":"(?<_VAL_2>[^"])"
"(?<_KEY_3>[^"])":"(?<_VAL_3>[^"])"

i want dynamic field extraction like this :
1. key1, key2, key3 .... is field name (because that key is changed dynamically)
2. value1, value2, value3 .... is field value

finally i want field extraction to simple.
use a one regular expression or delims or another way.
because that production is Maintenance by customer. so i must make easy to Maintenance.

I think there is a way like that use spath command in SPL
or another way.

thank you for answer

Re: How to use spath command in props.conf or transforms.conf?

leeyounsoo — Tue, 29 Sep 2020 19:12:44 GMT

i solve that like this :

transforms.conf

[my_stanza]
REGEX = "([^"]+)":(.?)"(.?)"
FORMAT = $1::$3
MV_ADD = True

thank you for answer.

Re: How to use spath command in props.conf or transforms.conf?

franktaylor — Thu, 20 Oct 2022 20:36:38 GMT

"(?<_KEY_1>[^"]*)":"(?<_VAL_1>[^"]*)"
how to get this to work where VAL value is not enclosed by quotes, such as this example :

networkConnect": {
            "localIp": "10.132.137.105",
            "localPort": 64248,
            "networkProtocol": "kIPv4",
            "remoteIp": "63.247.145.75",
            "remotePort": 88