Splunk Search

How to use spath command in props.conf or transforms.conf?

leeyounsoo
Path Finder

hello
I have a question

I have a data that access_log data and json data is mixed and my need is to extract field from mixed data.
splunk auto extraction field from that key=value format data
So, i received extracted json field(field name is "json") like this :

{"key":"value", "key":"value", "key":"value", "key":"value", "key":"value" ....(skip)} 

And if I search the json field, I return the data but, I want extracted field from JSON like this :

| fields key, key2, key3, key4 ....(skip)

Finally, I found spath command and I got the results that I wanted.
I tried to modify props.conf to automatically extract the field from json but it is not working.

What should I do?

ps: I tried t0 use the eval command like this :

| eval example = spath(json, 'json.key')

And like this:

| eval example = json.key 

But that is also not working.

0 Karma
1 Solution

leeyounsoo
Path Finder

i solve that like this :

  1. transforms.conf

    [my_stanza]
    REGEX = "([^"]+)":(.?)"(.?)"
    FORMAT = $1::$3
    MV_ADD = True

    thank you for answer.

View solution in original post

0 Karma

leeyounsoo
Path Finder

i solve that like this :

  1. transforms.conf

    [my_stanza]
    REGEX = "([^"]+)":(.?)"(.?)"
    FORMAT = $1::$3
    MV_ADD = True

    thank you for answer.

0 Karma

woodcock
Esteemed Legend

Go to Settings -> Fields -> Field extractoins -> New.
Enter anything that you like for Name (I suggest something like ColonCommaKVPs), Enter the exact name of your sourcetype in the named field, keep the default of Inline for Type and Sourcetype for Apply to, then enter this for Extraction/Transform:

"(?<_KEY_1>[^"]*)":"(?<_VAL_1>[^"]*)"

franktaylor
New Member

"(?<_KEY_1>[^"]*)":"(?<_VAL_1>[^"]*)"
how to get this to work where VAL value is not enclosed by quotes, such as this example :

networkConnect": {
            "localIp": "10.132.137.105",
            "localPort": 64248,
            "networkProtocol": "kIPv4",
            "remoteIp": "63.247.145.75",
            "remotePort": 88

0 Karma

leeyounsoo
Path Finder

thank you for answer.
but, that solution is not working.
I think there are two reasons.

  1. my data is some problem like this : 1-1) some data has zero string
  2. "json= {"key1":"value1", "key2":"","key3":"value3" .....}"

1-2) some data format is not "key":"value"
- value has blank before "value"
- "json= {"key1": "value1", "key2": "","key3": "value3" .....}"
- "key":(this place is blank)"value"

  1. props.conf overlap
  2. i write props.conf in $SPLUNK_HOME$/etc/system/local/props.conf
  3. your solution place in $SPLUNK_HOME$/etc/apps/$my_app$/local/props.conf
  4. I know that the .conf file under the system folder has a higher priority. So I moved EXTRACT-my = "(? <_ KEY_1> [^"] + ")" (? [^ "] +") to props.conf under the system folder. But it did not work.

What's wrong?

0 Karma

leeyounsoo
Path Finder

OK i will tried that.

thank you for answer!!

0 Karma

leeyounsoo
Path Finder

hello woodcock.
i tried that and that look like work, but that is static.(my think...
Perhaps, You gave me the answer that I wanted. but
Maybe it is that i did not understand)

so, In my opinion, your answer is need loop like this :
[report-stanza]
"(?<_KEY_1>[^"])":"(?<_VAL_1>[^"])"
"(?<_KEY_2>[^"])":"(?<_VAL_2>[^"])"
"(?<_KEY_3>[^"])":"(?<_VAL_3>[^"])"

i want dynamic field extraction like this :
1. key1, key2, key3 .... is field name (because that key is changed dynamically)
2. value1, value2, value3 .... is field value

finally i want field extraction to simple.
use a one regular expression or delims or another way.
because that production is Maintenance by customer. so i must make easy to Maintenance.

I think there is a way like that use spath command in SPL
or another way.

thank you for answer

0 Karma

woodcock
Esteemed Legend

You should do it exactly as i described (from the GUI) and make it work, then repackage it in whatever app. My solution, with a minor adjustment of changing the + characters to * will work for null values. I have modified my answer so it now reads like this:

"(?<_KEY_1>[^"]*)":"(?<_VAL_1>[^"]*)"
0 Karma

ssadanala1
Contributor

Please specify

INDEXED_EXTRACTIONS = JSON OR KV_MODE = json but not both .

The above should extract fields at index time for structured json data .

leeyounsoo
Path Finder

Sorry.

I think you gave me a slightly different answer because I did not give you the exact details.

The access_log and json data are mixed as follows:
"POST /sample.logs HTTP / 1.1" 200 0 "app: /data.swf ..." "another logs ...." 10.10.10.10 - - [November / 2018: 16: 31: "userAgent data. ... "" json = {"key": "value", "key": "value"

The json field is extracted by the automatic extraction setting of the splunk.
What I want is the value of the json field.

The search command was able to extract the fields with spath command like this :
.... | spath input = json.

How can I get the same results in props.conf or transforms.conf?

thank you for answer

0 Karma

niketn
Legend

Refer to documentation for KV_MODE : https://docs.splunk.com/Documentation/Splunk/latest/Knowledge/Automatickey-valuefieldextractionsatse...

and INDEXED_EXTRACTIONS: http://docs.splunk.com/Documentation/Splunk/latest/Data/Extractfieldsfromfileswithstructureddata#Pro...

____________________________________________
| makeresults | eval message= "Happy Splunking!!!"
0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...