Getting Data In

auto extract json

Cuyose
Builder

I have been researching this and came up with some odd ways of doing it, and about 40 other ways, none being the same. Given an even that looks like this in a sourcetype, what would my transforms and props configs be to get splunk to just recognize the key value pairs? Data obfuscated, but format and delimiters remain the same.

So I would love for splunk to recognize
fields:value
example
h=HK_SV_
v_m=DBG
h=h://hk-d.j.m/ds/115893
etc.

[2014-06-22 04:03:21.013892] {"h":"HK_SV_", "v_m":"DBG", "mssg": {"sv_":{"h":"h://hk-d.j.m/ds/115893","mhd":"S","qs":"{"d_f":{"d":{"d_d":1066325,"shg_ddss":{"fs_m":"k","s_m":"'","s1":"1450 s d","s2":,"y":"Bgm ","g":"f","sd":"94010","h":"650-245-4896","y_d":"S"},"df_bx_sk":"J-14","df__d":1,"s_wds_sf_d":0,"b_sk_gd":fs,"sk_s":" d  sv   my hs mh.","hs_smzd":fs,"":,"gf":fs,"wdw_mh":"J","wdw_y":2014,"ss":"SKD","d_ms":[]},"s":{"sm_d":"115893","mv_d":42628,"mv_ss":"V","_d":1,"shg_ddss":{"fs_m":"k","s_m":"'","s1":"1450 s d","s2":,"y":"Bgm ","g":"f","sd":"94010","h":"650-245-4896","y_d":"S"},"d_sk":fs,"wys__sk":,"_bg":,"d_shms":0,"g_":19.99,"sbs_m":1,"sy_f_d":"","s_mh":2,"s_y":2013,"ss_d":""}}}","ss":{"ss":"sss","d":{"d_f":{"d":{"d_d":1066325,"shg_ddss":{"fs_m":"k","s_m":"'","s1":"1450 s d","s2":,"y":"Bgm ","g":"f","sd":"94010","h":"650-245-4896","y_d":"S"},"df_bx_sk":"J-14","df__d":1,"s_wds_sf_d":0,"b_sk_gd":fs,"sk_s":" d  sv   my hs mh.","hs_smzd":fs,"":,"gf":fs,"wdw_mh":"J","wdw_y":2014,"ss":"SKD","d_ms":[]},"s":{"sm_d":"115893","mv_d":42628,"mv_ss":"V","_d":1,"shg_ddss":{"fs_m":"k","s_m":"'","s1":"1450 s d","s2":,"y":"Bgm ","g":"f","sd":"94010","h":"650-245-4896","y_d":"S"},"d_sk":fs,"wys__sk":,"_bg":,"d_shms":0,"g_":19.99,"sbs_m":1,"sy_f_d":"","s_mh":2,"s_y":2013,"ss_d":""}}}},"sd_m":0.074746}}, "sm_d": 115893, "mv_d": , "qs_d": }
0 Karma

martin_mueller
SplunkTrust
SplunkTrust

There were plenty more things wrong with the JSON, such as unquoted strings or empty values.

After fixing those you can do this:

| stats count | eval _raw = "[2014-06-22 04:03:21.013892] {
    \"h\": \"HK_SV_\",
    \"v_m\": \"DBG\",
    \"mssg\": {
        \"sv_\": {
            \"h\": \"h://hk-d.j.m/ds/115893\",
            \"mhd\": \"S\",
            \"qs\": {
                \"d_f\": {
                    \"d\": {
                        \"d_d\": 1066325,
                        \"shg_ddss\": {
                            \"fs_m\": \"k\",
                            \"s_m\": \"'\",
                            \"s1\": \"1450sd\",
                            \"s2\": \"\",
                            \"y\": \"Bgm\",
                            \"g\": \"f\",
                            \"sd\": \"94010\",
                            \"h\": \"650-245-4896\",
                            \"y_d\": \"S\"
                        },
                        \"df_bx_sk\": \"J-14\",
                        \"df__d\": 1,
                        \"s_wds_sf_d\": 0,
                        \"b_sk_gd\": \"fs\",
                        \"sk_s\": \"dsvmyhsmh.\",
                        \"hs_smzd\": \"fs\",
                        \"\": \"\",
                        \"gf\": \"fs\",
                        \"wdw_mh\": \"J\",
                        \"wdw_y\": 2014,
                        \"ss\": \"SKD\",
                        \"d_ms\": []
                    },
                    \"s\": {
                        \"sm_d\": \"115893\",
                        \"mv_d\": 42628,
                        \"mv_ss\": \"V\",
                        \"_d\": 1,
                        \"shg_ddss\": {
                            \"fs_m\": \"k\",
                            \"s_m\": \"'\",
                            \"s1\": \"1450sd\",
                            \"s2\": \"\",
                            \"y\": \"Bgm\",
                            \"g\": \"f\",
                            \"sd\": \"94010\",
                            \"h\": \"650-245-4896\",
                            \"y_d\": \"S\"
                        },
                        \"d_sk\": \"fs\",
                        \"wys__sk\": \"\",
                        \"_bg\": \"\",
                        \"d_shms\": 0,
                        \"g_\": 19.99,
                        \"sbs_m\": 1,
                        \"sy_f_d\": \"\",
                        \"s_mh\": 2,
                        \"s_y\": 2013,
                        \"ss_d\": \"\"
                    }
                }
            },
            \"ss\": {
                \"ss\": \"sss\",
                \"d\": {
                    \"d_f\": {
                        \"d\": {
                            \"d_d\": 1066325,
                            \"shg_ddss\": {
                                \"fs_m\": \"k\",
                                \"s_m\": \"'\",
                                \"s1\": \"1450sd\",
                                \"s2\": \"\",
                                \"y\": \"Bgm\",
                                \"g\": \"f\",
                                \"sd\": \"94010\",
                                \"h\": \"650-245-4896\",
                                \"y_d\": \"S\"
                            },
                            \"df_bx_sk\": \"J-14\",
                            \"df__d\": 1,
                            \"s_wds_sf_d\": 0,
                            \"b_sk_gd\": \"fs\",
                            \"sk_s\": \"dsvmyhsmh.\",
                            \"hs_smzd\": \"fs\",
                            \"\": \"\",
                            \"gf\": \"fs\",
                            \"wdw_mh\": \"J\",
                            \"wdw_y\": 2014,
                            \"ss\": \"SKD\",
                            \"d_ms\": []
                        },
                        \"s\": {
                            \"sm_d\": \"115893\",
                            \"mv_d\": 42628,
                            \"mv_ss\": \"V\",
                            \"_d\": 1,
                            \"shg_ddss\": {
                                \"fs_m\": \"k\",
                                \"s_m\": \"'\",
                                \"s1\": \"1450sd\",
                                \"s2\": \"\",
                                \"y\": \"Bgm\",
                                \"g\": \"f\",
                                \"sd\": \"94010\",
                                \"h\": \"650-245-4896\",
                                \"y_d\": \"S\"
                            },
                            \"d_sk\": \"fs\",
                            \"wys__sk\": \"\",
                            \"_bg\": \"\",
                            \"d_shms\": 0,
                            \"g_\": 19.99,
                            \"sbs_m\": 1,
                            \"sy_f_d\": \"\",
                            \"s_mh\": 2,
                            \"s_y\": 2013,
                            \"ss_d\": \"\"
                        }
                    }
                }
            },
            \"sd_m\": 0.074746
        }
    },
    \"sm_d\": 115893,
    \"mv_d\": \"\",
    \"qs_d\": \"\"
}"
| rex "(?s)^.{20}\.?\d*\]\s+(?<json>.*)$" | spath input=json

The key is extracting the field and passing it into spath. You'll get a boatload of fields afterwards.

0 Karma

Cuyose
Builder

It also appears the log event is not valid json, as this portion is wrapped in "'s that makes it invalid. Pretending those are removed though, what would the correct way to extract it be?

"{"d_f":{"d":{"d_d":10.........bg":,"d_shms":0,"g":19.99,"sbs_m":1,"sy_f_d":"","s_mh":2,"s_y":2013,"ss_d":""}}}"

0 Karma
Did you miss .conf21 Virtual?

Good news! The event's keynotes and many of its breakout sessions are now available online, and still totally FREE!