Getting Data In

auto extract json

Cuyose
Builder

I have been researching this and came up with some odd ways of doing it, and about 40 other ways, none being the same. Given an even that looks like this in a sourcetype, what would my transforms and props configs be to get splunk to just recognize the key value pairs? Data obfuscated, but format and delimiters remain the same.

So I would love for splunk to recognize
fields:value
example
h=HK_SV_
v_m=DBG
h=h://hk-d.j.m/ds/115893
etc.

[2014-06-22 04:03:21.013892] {"h":"HK_SV_", "v_m":"DBG", "mssg": {"sv_":{"h":"h://hk-d.j.m/ds/115893","mhd":"S","qs":"{"d_f":{"d":{"d_d":1066325,"shg_ddss":{"fs_m":"k","s_m":"'","s1":"1450 s d","s2":,"y":"Bgm ","g":"f","sd":"94010","h":"650-245-4896","y_d":"S"},"df_bx_sk":"J-14","df__d":1,"s_wds_sf_d":0,"b_sk_gd":fs,"sk_s":" d  sv   my hs mh.","hs_smzd":fs,"":,"gf":fs,"wdw_mh":"J","wdw_y":2014,"ss":"SKD","d_ms":[]},"s":{"sm_d":"115893","mv_d":42628,"mv_ss":"V","_d":1,"shg_ddss":{"fs_m":"k","s_m":"'","s1":"1450 s d","s2":,"y":"Bgm ","g":"f","sd":"94010","h":"650-245-4896","y_d":"S"},"d_sk":fs,"wys__sk":,"_bg":,"d_shms":0,"g_":19.99,"sbs_m":1,"sy_f_d":"","s_mh":2,"s_y":2013,"ss_d":""}}}","ss":{"ss":"sss","d":{"d_f":{"d":{"d_d":1066325,"shg_ddss":{"fs_m":"k","s_m":"'","s1":"1450 s d","s2":,"y":"Bgm ","g":"f","sd":"94010","h":"650-245-4896","y_d":"S"},"df_bx_sk":"J-14","df__d":1,"s_wds_sf_d":0,"b_sk_gd":fs,"sk_s":" d  sv   my hs mh.","hs_smzd":fs,"":,"gf":fs,"wdw_mh":"J","wdw_y":2014,"ss":"SKD","d_ms":[]},"s":{"sm_d":"115893","mv_d":42628,"mv_ss":"V","_d":1,"shg_ddss":{"fs_m":"k","s_m":"'","s1":"1450 s d","s2":,"y":"Bgm ","g":"f","sd":"94010","h":"650-245-4896","y_d":"S"},"d_sk":fs,"wys__sk":,"_bg":,"d_shms":0,"g_":19.99,"sbs_m":1,"sy_f_d":"","s_mh":2,"s_y":2013,"ss_d":""}}}},"sd_m":0.074746}}, "sm_d": 115893, "mv_d": , "qs_d": }
0 Karma

martin_mueller
SplunkTrust
SplunkTrust

There were plenty more things wrong with the JSON, such as unquoted strings or empty values.

After fixing those you can do this:

| stats count | eval _raw = "[2014-06-22 04:03:21.013892] {
    \"h\": \"HK_SV_\",
    \"v_m\": \"DBG\",
    \"mssg\": {
        \"sv_\": {
            \"h\": \"h://hk-d.j.m/ds/115893\",
            \"mhd\": \"S\",
            \"qs\": {
                \"d_f\": {
                    \"d\": {
                        \"d_d\": 1066325,
                        \"shg_ddss\": {
                            \"fs_m\": \"k\",
                            \"s_m\": \"'\",
                            \"s1\": \"1450sd\",
                            \"s2\": \"\",
                            \"y\": \"Bgm\",
                            \"g\": \"f\",
                            \"sd\": \"94010\",
                            \"h\": \"650-245-4896\",
                            \"y_d\": \"S\"
                        },
                        \"df_bx_sk\": \"J-14\",
                        \"df__d\": 1,
                        \"s_wds_sf_d\": 0,
                        \"b_sk_gd\": \"fs\",
                        \"sk_s\": \"dsvmyhsmh.\",
                        \"hs_smzd\": \"fs\",
                        \"\": \"\",
                        \"gf\": \"fs\",
                        \"wdw_mh\": \"J\",
                        \"wdw_y\": 2014,
                        \"ss\": \"SKD\",
                        \"d_ms\": []
                    },
                    \"s\": {
                        \"sm_d\": \"115893\",
                        \"mv_d\": 42628,
                        \"mv_ss\": \"V\",
                        \"_d\": 1,
                        \"shg_ddss\": {
                            \"fs_m\": \"k\",
                            \"s_m\": \"'\",
                            \"s1\": \"1450sd\",
                            \"s2\": \"\",
                            \"y\": \"Bgm\",
                            \"g\": \"f\",
                            \"sd\": \"94010\",
                            \"h\": \"650-245-4896\",
                            \"y_d\": \"S\"
                        },
                        \"d_sk\": \"fs\",
                        \"wys__sk\": \"\",
                        \"_bg\": \"\",
                        \"d_shms\": 0,
                        \"g_\": 19.99,
                        \"sbs_m\": 1,
                        \"sy_f_d\": \"\",
                        \"s_mh\": 2,
                        \"s_y\": 2013,
                        \"ss_d\": \"\"
                    }
                }
            },
            \"ss\": {
                \"ss\": \"sss\",
                \"d\": {
                    \"d_f\": {
                        \"d\": {
                            \"d_d\": 1066325,
                            \"shg_ddss\": {
                                \"fs_m\": \"k\",
                                \"s_m\": \"'\",
                                \"s1\": \"1450sd\",
                                \"s2\": \"\",
                                \"y\": \"Bgm\",
                                \"g\": \"f\",
                                \"sd\": \"94010\",
                                \"h\": \"650-245-4896\",
                                \"y_d\": \"S\"
                            },
                            \"df_bx_sk\": \"J-14\",
                            \"df__d\": 1,
                            \"s_wds_sf_d\": 0,
                            \"b_sk_gd\": \"fs\",
                            \"sk_s\": \"dsvmyhsmh.\",
                            \"hs_smzd\": \"fs\",
                            \"\": \"\",
                            \"gf\": \"fs\",
                            \"wdw_mh\": \"J\",
                            \"wdw_y\": 2014,
                            \"ss\": \"SKD\",
                            \"d_ms\": []
                        },
                        \"s\": {
                            \"sm_d\": \"115893\",
                            \"mv_d\": 42628,
                            \"mv_ss\": \"V\",
                            \"_d\": 1,
                            \"shg_ddss\": {
                                \"fs_m\": \"k\",
                                \"s_m\": \"'\",
                                \"s1\": \"1450sd\",
                                \"s2\": \"\",
                                \"y\": \"Bgm\",
                                \"g\": \"f\",
                                \"sd\": \"94010\",
                                \"h\": \"650-245-4896\",
                                \"y_d\": \"S\"
                            },
                            \"d_sk\": \"fs\",
                            \"wys__sk\": \"\",
                            \"_bg\": \"\",
                            \"d_shms\": 0,
                            \"g_\": 19.99,
                            \"sbs_m\": 1,
                            \"sy_f_d\": \"\",
                            \"s_mh\": 2,
                            \"s_y\": 2013,
                            \"ss_d\": \"\"
                        }
                    }
                }
            },
            \"sd_m\": 0.074746
        }
    },
    \"sm_d\": 115893,
    \"mv_d\": \"\",
    \"qs_d\": \"\"
}"
| rex "(?s)^.{20}\.?\d*\]\s+(?<json>.*)$" | spath input=json

The key is extracting the field and passing it into spath. You'll get a boatload of fields afterwards.

0 Karma

Cuyose
Builder

It also appears the log event is not valid json, as this portion is wrapped in "'s that makes it invalid. Pretending those are removed though, what would the correct way to extract it be?

"{"d_f":{"d":{"d_d":10.........bg":,"d_shms":0,"g":19.99,"sbs_m":1,"sy_f_d":"","s_mh":2,"s_y":2013,"ss_d":""}}}"

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...