https://community.splunk.com/t5/Getting-Data-In/auto-extract-json/m-p/114810#M24017 <P>I have been researching this and came up with some odd ways of doing it, and about 40 other ways, none being the same. Given an even that looks like this in a sourcetype, what would my transforms and props configs be to get splunk to just recognize the key value pairs? Data obfuscated, but format and delimiters remain the same.</P> <P>So I would love for splunk to recognize<BR /> fields:value<BR /> example<BR /> h=HK_SV_<BR /> v_m=DBG<BR /> h=h://hk-d.j.m/ds/115893<BR /> etc.</P> <PRE><CODE>[2014-06-22 04:03:21.013892] {"h":"HK_SV_", "v_m":"DBG", "mssg": {"sv_":{"h":"h://hk-d.j.m/ds/115893","mhd":"S","qs":"{"d_f":{"d":{"d_d":1066325,"shg_ddss":{"fs_m":"k","s_m":"'","s1":"1450 s d","s2":,"y":"Bgm ","g":"f","sd":"94010","h":"650-245-4896","y_d":"S"},"df_bx_sk":"J-14","df__d":1,"s_wds_sf_d":0,"b_sk_gd":fs,"sk_s":" d sv my hs mh.","hs_smzd":fs,"":,"gf":fs,"wdw_mh":"J","wdw_y":2014,"ss":"SKD","d_ms":[]},"s":{"sm_d":"115893","mv_d":42628,"mv_ss":"V","_d":1,"shg_ddss":{"fs_m":"k","s_m":"'","s1":"1450 s d","s2":,"y":"Bgm ","g":"f","sd":"94010","h":"650-245-4896","y_d":"S"},"d_sk":fs,"wys__sk":,"_bg":,"d_shms":0,"g_":19.99,"sbs_m":1,"sy_f_d":"","s_mh":2,"s_y":2013,"ss_d":""}}}","ss":{"ss":"sss","d":{"d_f":{"d":{"d_d":1066325,"shg_ddss":{"fs_m":"k","s_m":"'","s1":"1450 s d","s2":,"y":"Bgm ","g":"f","sd":"94010","h":"650-245-4896","y_d":"S"},"df_bx_sk":"J-14","df__d":1,"s_wds_sf_d":0,"b_sk_gd":fs,"sk_s":" d sv my hs mh.","hs_smzd":fs,"":,"gf":fs,"wdw_mh":"J","wdw_y":2014,"ss":"SKD","d_ms":[]},"s":{"sm_d":"115893","mv_d":42628,"mv_ss":"V","_d":1,"shg_ddss":{"fs_m":"k","s_m":"'","s1":"1450 s d","s2":,"y":"Bgm ","g":"f","sd":"94010","h":"650-245-4896","y_d":"S"},"d_sk":fs,"wys__sk":,"_bg":,"d_shms":0,"g_":19.99,"sbs_m":1,"sy_f_d":"","s_mh":2,"s_y":2013,"ss_d":""}}}},"sd_m":0.074746}}, "sm_d": 115893, "mv_d": , "qs_d": } </CODE></PRE> Mon, 28 Sep 2020 16:54:01 GMT Cuyose 2020-09-28T16:54:01Z https://community.splunk.com/t5/Getting-Data-In/auto-extract-json/m-p/114810#M24017 <P>I have been researching this and came up with some odd ways of doing it, and about 40 other ways, none being the same. Given an even that looks like this in a sourcetype, what would my transforms and props configs be to get splunk to just recognize the key value pairs? Data obfuscated, but format and delimiters remain the same.</P> <P>So I would love for splunk to recognize<BR /> fields:value<BR /> example<BR /> h=HK_SV_<BR /> v_m=DBG<BR /> h=h://hk-d.j.m/ds/115893<BR /> etc.</P> <PRE><CODE>[2014-06-22 04:03:21.013892] {"h":"HK_SV_", "v_m":"DBG", "mssg": {"sv_":{"h":"h://hk-d.j.m/ds/115893","mhd":"S","qs":"{"d_f":{"d":{"d_d":1066325,"shg_ddss":{"fs_m":"k","s_m":"'","s1":"1450 s d","s2":,"y":"Bgm ","g":"f","sd":"94010","h":"650-245-4896","y_d":"S"},"df_bx_sk":"J-14","df__d":1,"s_wds_sf_d":0,"b_sk_gd":fs,"sk_s":" d sv my hs mh.","hs_smzd":fs,"":,"gf":fs,"wdw_mh":"J","wdw_y":2014,"ss":"SKD","d_ms":[]},"s":{"sm_d":"115893","mv_d":42628,"mv_ss":"V","_d":1,"shg_ddss":{"fs_m":"k","s_m":"'","s1":"1450 s d","s2":,"y":"Bgm ","g":"f","sd":"94010","h":"650-245-4896","y_d":"S"},"d_sk":fs,"wys__sk":,"_bg":,"d_shms":0,"g_":19.99,"sbs_m":1,"sy_f_d":"","s_mh":2,"s_y":2013,"ss_d":""}}}","ss":{"ss":"sss","d":{"d_f":{"d":{"d_d":1066325,"shg_ddss":{"fs_m":"k","s_m":"'","s1":"1450 s d","s2":,"y":"Bgm ","g":"f","sd":"94010","h":"650-245-4896","y_d":"S"},"df_bx_sk":"J-14","df__d":1,"s_wds_sf_d":0,"b_sk_gd":fs,"sk_s":" d sv my hs mh.","hs_smzd":fs,"":,"gf":fs,"wdw_mh":"J","wdw_y":2014,"ss":"SKD","d_ms":[]},"s":{"sm_d":"115893","mv_d":42628,"mv_ss":"V","_d":1,"shg_ddss":{"fs_m":"k","s_m":"'","s1":"1450 s d","s2":,"y":"Bgm ","g":"f","sd":"94010","h":"650-245-4896","y_d":"S"},"d_sk":fs,"wys__sk":,"_bg":,"d_shms":0,"g_":19.99,"sbs_m":1,"sy_f_d":"","s_mh":2,"s_y":2013,"ss_d":""}}}},"sd_m":0.074746}}, "sm_d": 115893, "mv_d": , "qs_d": } </CODE></PRE> Mon, 28 Sep 2020 16:54:01 GMT https://community.splunk.com/t5/Getting-Data-In/auto-extract-json/m-p/114810#M24017 Cuyose 2020-09-28T16:54:01Z https://community.splunk.com/t5/Getting-Data-In/auto-extract-json/m-p/114811#M24018 <P>It also appears the log event is not valid json, as this portion is wrapped in "'s that makes it invalid. Pretending those are removed though, what would the correct way to extract it be?</P> <P>"{"d_f":{"d":{"d_d":10.........bg":,"d_shms":0,"g":19.99,"sbs_m":1,"sy_f_d":"","s_mh":2,"s_y":2013,"ss_d":""}}}"</P> Mon, 28 Sep 2020 16:54:03 GMT https://community.splunk.com/t5/Getting-Data-In/auto-extract-json/m-p/114811#M24018 Cuyose 2020-09-28T16:54:03Z https://community.splunk.com/t5/Getting-Data-In/auto-extract-json/m-p/114812#M24019 <P>There were plenty more things wrong with the JSON, such as unquoted strings or empty values.</P> <P>After fixing those you can do this:</P> <PRE><CODE>| stats count | eval _raw = "[2014-06-22 04:03:21.013892] { \"h\": \"HK_SV_\", \"v_m\": \"DBG\", \"mssg\": { \"sv_\": { \"h\": \"h://hk-d.j.m/ds/115893\", \"mhd\": \"S\", \"qs\": { \"d_f\": { \"d\": { \"d_d\": 1066325, \"shg_ddss\": { \"fs_m\": \"k\", \"s_m\": \"'\", \"s1\": \"1450sd\", \"s2\": \"\", \"y\": \"Bgm\", \"g\": \"f\", \"sd\": \"94010\", \"h\": \"650-245-4896\", \"y_d\": \"S\" }, \"df_bx_sk\": \"J-14\", \"df__d\": 1, \"s_wds_sf_d\": 0, \"b_sk_gd\": \"fs\", \"sk_s\": \"dsvmyhsmh.\", \"hs_smzd\": \"fs\", \"\": \"\", \"gf\": \"fs\", \"wdw_mh\": \"J\", \"wdw_y\": 2014, \"ss\": \"SKD\", \"d_ms\": [] }, \"s\": { \"sm_d\": \"115893\", \"mv_d\": 42628, \"mv_ss\": \"V\", \"_d\": 1, \"shg_ddss\": { \"fs_m\": \"k\", \"s_m\": \"'\", \"s1\": \"1450sd\", \"s2\": \"\", \"y\": \"Bgm\", \"g\": \"f\", \"sd\": \"94010\", \"h\": \"650-245-4896\", \"y_d\": \"S\" }, \"d_sk\": \"fs\", \"wys__sk\": \"\", \"_bg\": \"\", \"d_shms\": 0, \"g_\": 19.99, \"sbs_m\": 1, \"sy_f_d\": \"\", \"s_mh\": 2, \"s_y\": 2013, \"ss_d\": \"\" } } }, \"ss\": { \"ss\": \"sss\", \"d\": { \"d_f\": { \"d\": { \"d_d\": 1066325, \"shg_ddss\": { \"fs_m\": \"k\", \"s_m\": \"'\", \"s1\": \"1450sd\", \"s2\": \"\", \"y\": \"Bgm\", \"g\": \"f\", \"sd\": \"94010\", \"h\": \"650-245-4896\", \"y_d\": \"S\" }, \"df_bx_sk\": \"J-14\", \"df__d\": 1, \"s_wds_sf_d\": 0, \"b_sk_gd\": \"fs\", \"sk_s\": \"dsvmyhsmh.\", \"hs_smzd\": \"fs\", \"\": \"\", \"gf\": \"fs\", \"wdw_mh\": \"J\", \"wdw_y\": 2014, \"ss\": \"SKD\", \"d_ms\": [] }, \"s\": { \"sm_d\": \"115893\", \"mv_d\": 42628, \"mv_ss\": \"V\", \"_d\": 1, \"shg_ddss\": { \"fs_m\": \"k\", \"s_m\": \"'\", \"s1\": \"1450sd\", \"s2\": \"\", \"y\": \"Bgm\", \"g\": \"f\", \"sd\": \"94010\", \"h\": \"650-245-4896\", \"y_d\": \"S\" }, \"d_sk\": \"fs\", \"wys__sk\": \"\", \"_bg\": \"\", \"d_shms\": 0, \"g_\": 19.99, \"sbs_m\": 1, \"sy_f_d\": \"\", \"s_mh\": 2, \"s_y\": 2013, \"ss_d\": \"\" } } } }, \"sd_m\": 0.074746 } }, \"sm_d\": 115893, \"mv_d\": \"\", \"qs_d\": \"\" }" | rex "(?s)^.{20}\.?\d*\]\s+(?&lt;json&gt;.*)$" | spath input=json </CODE></PRE> <P>The key is extracting the field and passing it into <CODE>spath</CODE>. You'll get a boatload of fields afterwards.</P> Thu, 26 Jun 2014 11:02:40 GMT https://community.splunk.com/t5/Getting-Data-In/auto-extract-json/m-p/114812#M24019 martin_mueller 2014-06-26T11:02:40Z