auto extract json

[Cuyose]

I have been researching this and came up with some odd ways of doing it, and about 40 other ways, none being the same. Given an even that looks like this in a sourcetype, what would my transforms and props configs be to get splunk to just recognize the key value pairs? Data obfuscated, but format and delimiters remain the same.

So I would love for splunk to recognize
fields:value
example
h=HK_SV_
v_m=DBG
h=h://hk-d.j.m/ds/115893
etc.

[2014-06-22 04:03:21.013892] {"h":"HK_SV_", "v_m":"DBG", "mssg": {"sv_":{"h":"h://hk-d.j.m/ds/115893","mhd":"S","qs":"{"d_f":{"d":{"d_d":1066325,"shg_ddss":{"fs_m":"k","s_m":"'","s1":"1450 s d","s2":,"y":"Bgm ","g":"f","sd":"94010","h":"650-245-4896","y_d":"S"},"df_bx_sk":"J-14","df__d":1,"s_wds_sf_d":0,"b_sk_gd":fs,"sk_s":" d  sv   my hs mh.","hs_smzd":fs,"":,"gf":fs,"wdw_mh":"J","wdw_y":2014,"ss":"SKD","d_ms":[]},"s":{"sm_d":"115893","mv_d":42628,"mv_ss":"V","_d":1,"shg_ddss":{"fs_m":"k","s_m":"'","s1":"1450 s d","s2":,"y":"Bgm ","g":"f","sd":"94010","h":"650-245-4896","y_d":"S"},"d_sk":fs,"wys__sk":,"_bg":,"d_shms":0,"g_":19.99,"sbs_m":1,"sy_f_d":"","s_mh":2,"s_y":2013,"ss_d":""}}}","ss":{"ss":"sss","d":{"d_f":{"d":{"d_d":1066325,"shg_ddss":{"fs_m":"k","s_m":"'","s1":"1450 s d","s2":,"y":"Bgm ","g":"f","sd":"94010","h":"650-245-4896","y_d":"S"},"df_bx_sk":"J-14","df__d":1,"s_wds_sf_d":0,"b_sk_gd":fs,"sk_s":" d  sv   my hs mh.","hs_smzd":fs,"":,"gf":fs,"wdw_mh":"J","wdw_y":2014,"ss":"SKD","d_ms":[]},"s":{"sm_d":"115893","mv_d":42628,"mv_ss":"V","_d":1,"shg_ddss":{"fs_m":"k","s_m":"'","s1":"1450 s d","s2":,"y":"Bgm ","g":"f","sd":"94010","h":"650-245-4896","y_d":"S"},"d_sk":fs,"wys__sk":,"_bg":,"d_shms":0,"g_":19.99,"sbs_m":1,"sy_f_d":"","s_mh":2,"s_y":2013,"ss_d":""}}}},"sd_m":0.074746}}, "sm_d": 115893, "mv_d": , "qs_d": }

[martin_mueller]

There were plenty more things wrong with the JSON, such as unquoted strings or empty values.

After fixing those you can do this:

| stats count | eval _raw = "[2014-06-22 04:03:21.013892] {
    \"h\": \"HK_SV_\",
    \"v_m\": \"DBG\",
    \"mssg\": {
        \"sv_\": {
            \"h\": \"h://hk-d.j.m/ds/115893\",
            \"mhd\": \"S\",
            \"qs\": {
                \"d_f\": {
                    \"d\": {
                        \"d_d\": 1066325,
                        \"shg_ddss\": {
                            \"fs_m\": \"k\",
                            \"s_m\": \"'\",
                            \"s1\": \"1450sd\",
                            \"s2\": \"\",
                            \"y\": \"Bgm\",
                            \"g\": \"f\",
                            \"sd\": \"94010\",
                            \"h\": \"650-245-4896\",
                            \"y_d\": \"S\"
                        },
                        \"df_bx_sk\": \"J-14\",
                        \"df__d\": 1,
                        \"s_wds_sf_d\": 0,
                        \"b_sk_gd\": \"fs\",
                        \"sk_s\": \"dsvmyhsmh.\",
                        \"hs_smzd\": \"fs\",
                        \"\": \"\",
                        \"gf\": \"fs\",
                        \"wdw_mh\": \"J\",
                        \"wdw_y\": 2014,
                        \"ss\": \"SKD\",
                        \"d_ms\": []
                    },
                    \"s\": {
                        \"sm_d\": \"115893\",
                        \"mv_d\": 42628,
                        \"mv_ss\": \"V\",
                        \"_d\": 1,
                        \"shg_ddss\": {
                            \"fs_m\": \"k\",
                            \"s_m\": \"'\",
                            \"s1\": \"1450sd\",
                            \"s2\": \"\",
                            \"y\": \"Bgm\",
                            \"g\": \"f\",
                            \"sd\": \"94010\",
                            \"h\": \"650-245-4896\",
                            \"y_d\": \"S\"
                        },
                        \"d_sk\": \"fs\",
                        \"wys__sk\": \"\",
                        \"_bg\": \"\",
                        \"d_shms\": 0,
                        \"g_\": 19.99,
                        \"sbs_m\": 1,
                        \"sy_f_d\": \"\",
                        \"s_mh\": 2,
                        \"s_y\": 2013,
                        \"ss_d\": \"\"
                    }
                }
            },
            \"ss\": {
                \"ss\": \"sss\",
                \"d\": {
                    \"d_f\": {
                        \"d\": {
                            \"d_d\": 1066325,
                            \"shg_ddss\": {
                                \"fs_m\": \"k\",
                                \"s_m\": \"'\",
                                \"s1\": \"1450sd\",
                                \"s2\": \"\",
                                \"y\": \"Bgm\",
                                \"g\": \"f\",
                                \"sd\": \"94010\",
                                \"h\": \"650-245-4896\",
                                \"y_d\": \"S\"
                            },
                            \"df_bx_sk\": \"J-14\",
                            \"df__d\": 1,
                            \"s_wds_sf_d\": 0,
                            \"b_sk_gd\": \"fs\",
                            \"sk_s\": \"dsvmyhsmh.\",
                            \"hs_smzd\": \"fs\",
                            \"\": \"\",
                            \"gf\": \"fs\",
                            \"wdw_mh\": \"J\",
                            \"wdw_y\": 2014,
                            \"ss\": \"SKD\",
                            \"d_ms\": []
                        },
                        \"s\": {
                            \"sm_d\": \"115893\",
                            \"mv_d\": 42628,
                            \"mv_ss\": \"V\",
                            \"_d\": 1,
                            \"shg_ddss\": {
                                \"fs_m\": \"k\",
                                \"s_m\": \"'\",
                                \"s1\": \"1450sd\",
                                \"s2\": \"\",
                                \"y\": \"Bgm\",
                                \"g\": \"f\",
                                \"sd\": \"94010\",
                                \"h\": \"650-245-4896\",
                                \"y_d\": \"S\"
                            },
                            \"d_sk\": \"fs\",
                            \"wys__sk\": \"\",
                            \"_bg\": \"\",
                            \"d_shms\": 0,
                            \"g_\": 19.99,
                            \"sbs_m\": 1,
                            \"sy_f_d\": \"\",
                            \"s_mh\": 2,
                            \"s_y\": 2013,
                            \"ss_d\": \"\"
                        }
                    }
                }
            },
            \"sd_m\": 0.074746
        }
    },
    \"sm_d\": 115893,
    \"mv_d\": \"\",
    \"qs_d\": \"\"
}"
| rex "(?s)^.{20}\.?\d*\]\s+(?<json>.*)$" | spath input=json

The key is extracting the field and passing it into spath. You'll get a boatload of fields afterwards.

[Cuyose]

It also appears the log event is not valid json, as this portion is wrapped in "'s that makes it invalid. Pretending those are removed though, what would the correct way to extract it be?

"{"d_f":{"d":{"d_d":10.........bg":,"d_shms":0,"g":19.99,"sbs_m":1,"sy_f_d":"","s_mh":2,"s_y":2013,"ss_d":""}}}"