Splunk Search

How to use spath command in props.conf or transforms.conf?

leeyounsoo
Path Finder

hello
I have a question

I have a data that access_log data and json data is mixed and my need is to extract field from mixed data.
splunk auto extraction field from that key=value format data
So, i received extracted json field(field name is "json") like this :

{"key":"value", "key":"value", "key":"value", "key":"value", "key":"value" ....(skip)} 

And if I search the json field, I return the data but, I want extracted field from JSON like this :

| fields key, key2, key3, key4 ....(skip)

Finally, I found spath command and I got the results that I wanted.
I tried to modify props.conf to automatically extract the field from json but it is not working.

What should I do?

ps: I tried t0 use the eval command like this :

| eval example = spath(json, 'json.key')

And like this:

| eval example = json.key 

But that is also not working.

0 Karma
1 Solution

leeyounsoo
Path Finder

i solve that like this :

  1. transforms.conf

    [my_stanza]
    REGEX = "([^"]+)":(.?)"(.?)"
    FORMAT = $1::$3
    MV_ADD = True

    thank you for answer.

View solution in original post

0 Karma

leeyounsoo
Path Finder

i solve that like this :

  1. transforms.conf

    [my_stanza]
    REGEX = "([^"]+)":(.?)"(.?)"
    FORMAT = $1::$3
    MV_ADD = True

    thank you for answer.

0 Karma

woodcock
Esteemed Legend

Go to Settings -> Fields -> Field extractoins -> New.
Enter anything that you like for Name (I suggest something like ColonCommaKVPs), Enter the exact name of your sourcetype in the named field, keep the default of Inline for Type and Sourcetype for Apply to, then enter this for Extraction/Transform:

"(?<_KEY_1>[^"]*)":"(?<_VAL_1>[^"]*)"

leeyounsoo
Path Finder

thank you for answer.
but, that solution is not working.
I think there are two reasons.

  1. my data is some problem like this : 1-1) some data has zero string
  2. "json= {"key1":"value1", "key2":"","key3":"value3" .....}"

1-2) some data format is not "key":"value"
- value has blank before "value"
- "json= {"key1": "value1", "key2": "","key3": "value3" .....}"
- "key":(this place is blank)"value"

  1. props.conf overlap
  2. i write props.conf in $SPLUNK_HOME$/etc/system/local/props.conf
  3. your solution place in $SPLUNK_HOME$/etc/apps/$my_app$/local/props.conf
  4. I know that the .conf file under the system folder has a higher priority. So I moved EXTRACT-my = "(? <_ KEY_1> [^"] + ")" (? [^ "] +") to props.conf under the system folder. But it did not work.

What's wrong?

0 Karma

leeyounsoo
Path Finder

OK i will tried that.

thank you for answer!!

0 Karma

leeyounsoo
Path Finder

hello woodcock.
i tried that and that look like work, but that is static.(my think...
Perhaps, You gave me the answer that I wanted. but
Maybe it is that i did not understand)

so, In my opinion, your answer is need loop like this :
[report-stanza]
"(?<_KEY_1>[^"])":"(?<_VAL_1>[^"])"
"(?<_KEY_2>[^"])":"(?<_VAL_2>[^"])"
"(?<_KEY_3>[^"])":"(?<_VAL_3>[^"])"

i want dynamic field extraction like this :
1. key1, key2, key3 .... is field name (because that key is changed dynamically)
2. value1, value2, value3 .... is field value

finally i want field extraction to simple.
use a one regular expression or delims or another way.
because that production is Maintenance by customer. so i must make easy to Maintenance.

I think there is a way like that use spath command in SPL
or another way.

thank you for answer

0 Karma

woodcock
Esteemed Legend

You should do it exactly as i described (from the GUI) and make it work, then repackage it in whatever app. My solution, with a minor adjustment of changing the + characters to * will work for null values. I have modified my answer so it now reads like this:

"(?<_KEY_1>[^"]*)":"(?<_VAL_1>[^"]*)"
0 Karma

ssadanala1
Contributor

Please specify

INDEXED_EXTRACTIONS = JSON OR KV_MODE = json but not both .

The above should extract fields at index time for structured json data .

leeyounsoo
Path Finder

Sorry.

I think you gave me a slightly different answer because I did not give you the exact details.

The access_log and json data are mixed as follows:
"POST /sample.logs HTTP / 1.1" 200 0 "app: /data.swf ..." "another logs ...." 10.10.10.10 - - [November / 2018: 16: 31: "userAgent data. ... "" json = {"key": "value", "key": "value"

The json field is extracted by the automatic extraction setting of the splunk.
What I want is the value of the json field.

The search command was able to extract the fields with spath command like this :
.... | spath input = json.

How can I get the same results in props.conf or transforms.conf?

thank you for answer

0 Karma

Refer to documentation for KV_MODE : https://docs.splunk.com/Documentation/Splunk/latest/Knowledge/Automatickey-valuefieldextractionsatse...

and INDEXED_EXTRACTIONS: http://docs.splunk.com/Documentation/Splunk/latest/Data/Extractfieldsfromfileswithstructureddata#Pro...

____________________________________________
| makeresults | eval message= "Happy Splunking!!!"
0 Karma
Get Updates on the Splunk Community!

Introducing Ingest Actions: Filter, Mask, Route, Repeat

WATCH NOW Ingest Actions (IA) is the best new way to easily filter, mask and route your data in Splunk® ...

Splunk Forwarders and Forced Time Based Load Balancing

Splunk customers use universal forwarders to collect and send data to Splunk. A universal forwarder can send ...

NEW! Log Views in Splunk Observability Dashboards Gives Context From a Single Page

Today, Splunk Observability releases log views, a new feature for users to add their logs data from Splunk Log ...