How to use spath command in props.conf or transforms.conf?

[leeyounsoo]

hello
I have a question

I have a data that access_log data and json data is mixed and my need is to extract field from mixed data.
splunk auto extraction field from that key=value format data
So, i received extracted json field(field name is "json") like this :

{"key":"value", "key":"value", "key":"value", "key":"value", "key":"value" ....(skip)} 

And if I search the json field, I return the data but, I want extracted field from JSON like this :

| fields key, key2, key3, key4 ....(skip)

Finally, I found spath command and I got the results that I wanted.
I tried to modify props.conf to automatically extract the field from json but it is not working.

What should I do?

ps: I tried t0 use the eval command like this :

| eval example = spath(json, 'json.key')

And like this:

| eval example = json.key 

But that is also not working.

[leeyounsoo · Accepted solution]

i solve that like this :

1. transforms.conf

   [my_stanza]
   REGEX = "([^"]+)":(.?)"(.?)"
   FORMAT = $1::$3
   MV_ADD = True

   thank you for answer.

[woodcock]

Go to Settings -> Fields -> Field extractoins -> New.
Enter anything that you like for Name (I suggest something like ColonCommaKVPs), Enter the exact name of your sourcetype in the named field, keep the default of Inline for Type and Sourcetype for Apply to, then enter this for Extraction/Transform:

"(?<_KEY_1>[^"]*)":"(?<_VAL_1>[^"]*)"

[franktaylor]

"(?<_KEY_1>[^"]*)":"(?<_VAL_1>[^"]*)"
how to get this to work where VAL value is not enclosed by quotes, such as this example :

networkConnect": {
            "localIp": "10.132.137.105",
            "localPort": 64248,
            "networkProtocol": "kIPv4",
            "remoteIp": "63.247.145.75",
            "remotePort": 88

[leeyounsoo]

thank you for answer.
but, that solution is not working.
I think there are two reasons.

1. my data is some problem like this : 1-1) some data has zero string
2. "json= {"key1":"value1", "key2":"","key3":"value3" .....}"

1-2) some data format is not "key":"value"
- value has blank before "value"
- "json= {"key1": "value1", "key2": "","key3": "value3" .....}"
- "key":(this place is blank)"value"

1. props.conf overlap
2. i write props.conf in $SPLUNK_HOME$/etc/system/local/props.conf
3. your solution place in $SPLUNK_HOME$/etc/apps/$my_app$/local/props.conf
4. I know that the .conf file under the system folder has a higher priority. So I moved EXTRACT-my = "(? <_ KEY_1> [^"] + ")" (? [^ "] +") to props.conf under the system folder. But it did not work.

What's wrong?

[leeyounsoo]

OK i will tried that.

thank you for answer!!

[leeyounsoo]

hello woodcock.
i tried that and that look like work, but that is static.(my think...
Perhaps, You gave me the answer that I wanted. but
Maybe it is that i did not understand)

so, In my opinion, your answer is need loop like this :
[report-stanza]
"(?<_KEY_1>[^"])":"(?<_VAL_1>[^"])"
"(?<_KEY_2>[^"])":"(?<_VAL_2>[^"])"
"(?<_KEY_3>[^"])":"(?<_VAL_3>[^"])"

i want dynamic field extraction like this :
1. key1, key2, key3 .... is field name (because that key is changed dynamically)
2. value1, value2, value3 .... is field value

finally i want field extraction to simple.
use a one regular expression or delims or another way.
because that production is Maintenance by customer. so i must make easy to Maintenance.

I think there is a way like that use spath command in SPL
or another way.

thank you for answer

[woodcock]

You should do it exactly as i described (from the GUI) and make it work, then repackage it in whatever app. My solution, with a minor adjustment of changing the + characters to * will work for null values. I have modified my answer so it now reads like this:

"(?<_KEY_1>[^"]*)":"(?<_VAL_1>[^"]*)"

[ssadanala1]

Please specify

INDEXED_EXTRACTIONS = JSON OR KV_MODE = json but not both .

The above should extract fields at index time for structured json data .

[leeyounsoo]

Sorry.

I think you gave me a slightly different answer because I did not give you the exact details.

The access_log and json data are mixed as follows:
"POST /sample.logs HTTP / 1.1" 200 0 "app: /data.swf ..." "another logs ...." 10.10.10.10 - - [November / 2018: 16: 31: "userAgent data. ... "" json = {"key": "value", "key": "value"

The json field is extracted by the automatic extraction setting of the splunk.
What I want is the value of the json field.

The search command was able to extract the fields with spath command like this :
.... | spath input = json.

How can I get the same results in props.conf or transforms.conf?

thank you for answer

[niketn]

Refer to documentation for KV_MODE : https://docs.splunk.com/Documentation/Splunk/latest/Knowledge/Automatickey-valuefieldextractionsatse...

and INDEXED_EXTRACTIONS: http://docs.splunk.com/Documentation/Splunk/latest/Data/Extractfieldsfromfileswithstructureddata#Pro...