topic Field extraction json format in Splunk Search

Field extraction json format

splunklearner — Mon, 11 Nov 2024 13:44:24 GMT

Please help me to get these logs in a way that it provides all the fields please...

Nov 9 17:34:28 128.160.82.28 [local0.warning] <132>1 2024-11-09T17:34:28.436542Z AviVantage v-epswafhic2-wdc.hc.cloud.uk.hc-443 NILVALUE NILVALUE - {"adf":true,"significant":0,"udf":false,"virtualservice":"virtualservice-4583863f-48a3-42b9-8115-252a7fb487f5","report_timestamp":"2024-11-09T17:34:28.436542Z","service_engine":"GB-DRN-AB-Tier2-se-vxeuz","vcpu_id":0,"log_id":10181,"client_ip":"128.12.73.92","client_src_port":44908,"client_dest_port":443,"client_rtt":1,"http_version":"1.1","method":"HEAD","uri_path":"/path/to/monitor/page/","host":"udg1704n01.hc.cloud.uk.hc","response_content_type":"text/html","request_length":93,"response_length":94,"response_code":400,"response_time_first_byte":1,"response_time_last_byte":1,"compression_percentage":0,"compression":"","client_insights":"","request_headers":3,"response_headers":12,"request_state":"AVI_HTTP_REQUEST_STATE_READ_CLIENT_REQ_HDR","significant_log":["ADF_HTTP_BAD_REQUEST_PLAIN_HTTP_REQUEST_SENT_ON_HTTPS_PORT","ADF_RESPONSE_CODE_4XX"],"vs_ip":"128.160.71.14","request_id":"61e-RDl6-OZgZ","max_ingress_latency_fe":0,"avg_ingress_latency_fe":0,"conn_est_time_fe":1,"source_ip":"128.12.73.92","vs_name":"v-epswafhic2-wdc.hc.cloud.uk.hc-443","tenant_name":"admin"}

Re: Field extraction json format

ITWhisperer — Mon, 11 Nov 2024 13:35:56 GMT

The sample event that you posted does not contain valid json. I presume this is a copy/paste error or other typo. Please repost the raw data from your event (anonymised as required) in a code block (using the </> button above) to preserve formatting details.

Re: Field extraction json format

splunklearner — Mon, 11 Nov 2024 13:44:48 GMT

Hi,

please check now

Re: Field extraction json format

ITWhisperer — Mon, 11 Nov 2024 14:07:54 GMT

| rex "(?<json>\{.*\})" | spath input=json

Re: Field extraction json format

splunklearner — Mon, 11 Nov 2024 14:50:40 GMT

Thank you. It worked. One small doubt, will it be worked for upcoming new events also right? Is there any way to hide this in search rather than creating macro?

and can we do it during on-boarding itself during index or search time extraction? Please help me

Re: Field extraction json format

splunklearner — Mon, 11 Nov 2024 15:15:09 GMT

We are having 3 indexers with 2 cluster managers and 3 SH with one Deployer. its multi site cluster. Please help me to configure this setting before on-boarding rather than spath command? Please tell me in detail how to perform?

Re: Field extraction json format

PickleRick — Mon, 11 Nov 2024 18:00:27 GMT

Unfortunately, for now Splunk cannot perform a structured data extraction if the whole event is not a structured data (in other words - if you have a json or XML data which has some header, like in your example, Splunk cannot automatically extract data from it).

There is an idea about it at https://ideas.splunk.com/ideas/EID-I-208 - while it's already as "Future Prospect", you can give your vote to show your support for it.

At the moment the only thing you could do would be to cut the whole header away with SEDCMD during ingestion so that all that's left is a valid json structure. But that's not always what you want.

Re: Field extraction json format

splunklearner — Mon, 11 Nov 2024 18:04:02 GMT

Hi @PickleRick ,

Then what is use of KV_MODE = json that needs to be given in props.conf (saw somewhere a while ago).

Please let me understand whether my data contains both json and xml or only json? Because when i am using spath command provided by @ITWhisperer it extracted the fields... is it wrong? (if json and xml both present in my example event)

any idea on this?

Re: Field extraction json format

PickleRick — Mon, 11 Nov 2024 18:19:23 GMT

I mean that KV_MODE=something works only when the _whole event_ is just a blob of structured data. Without any additional parts to it.

So KV_MODE=json will work if your whole even consists of

{"my":"data","is":"json"}

but will not work if it's

<144>2014-11-11 11:23 Some lousy[24]: pseudo-syslog header with {"json":"data","further":"down","the":street"}

Re: Field extraction json format

splunklearner — Mon, 11 Nov 2024 18:41:14 GMT

| rex "(?<json>\{.*\})"
| spath input=json

so the above command works fine right for mixed pattern (json and xml) for my example? currently and for upcoming events? is there any other way to hide this query apart from macro?

Re: Field extraction json format

splunklearner — Tue, 12 Nov 2024 08:55:07 GMT

Appreciated @PickleRick and @ITWhisperer . Please answer my last question

| rex "(?<json>\{.*\})"
| spath input=json

so the above command works fine right for mixed pattern (json and xml) for my example? currently and for upcoming events? is there any other way to hide this query apart from macro?