Field extraction json format

[splunklearner]

Please help me to get these logs in a way that it provides all the fields please...

Nov 9 17:34:28 128.160.82.28 [local0.warning] <132>1 2024-11-09T17:34:28.436542Z AviVantage v-epswafhic2-wdc.hc.cloud.uk.hc-443 NILVALUE NILVALUE - {"adf":true,"significant":0,"udf":false,"virtualservice":"virtualservice-4583863f-48a3-42b9-8115-252a7fb487f5","report_timestamp":"2024-11-09T17:34:28.436542Z","service_engine":"GB-DRN-AB-Tier2-se-vxeuz","vcpu_id":0,"log_id":10181,"client_ip":"128.12.73.92","client_src_port":44908,"client_dest_port":443,"client_rtt":1,"http_version":"1.1","method":"HEAD","uri_path":"/path/to/monitor/page/","host":"udg1704n01.hc.cloud.uk.hc","response_content_type":"text/html","request_length":93,"response_length":94,"response_code":400,"response_time_first_byte":1,"response_time_last_byte":1,"compression_percentage":0,"compression":"","client_insights":"","request_headers":3,"response_headers":12,"request_state":"AVI_HTTP_REQUEST_STATE_READ_CLIENT_REQ_HDR","significant_log":["ADF_HTTP_BAD_REQUEST_PLAIN_HTTP_REQUEST_SENT_ON_HTTPS_PORT","ADF_RESPONSE_CODE_4XX"],"vs_ip":"128.160.71.14","request_id":"61e-RDl6-OZgZ","max_ingress_latency_fe":0,"avg_ingress_latency_fe":0,"conn_est_time_fe":1,"source_ip":"128.12.73.92","vs_name":"v-epswafhic2-wdc.hc.cloud.uk.hc-443","tenant_name":"admin"}

[splunklearner]

Appreciated @PickleRick and @ITWhisperer . Please answer my last question

| rex "(?<json>\{.*\})"
| spath input=json​

so the above command works fine right for mixed pattern (json and xml) for my example? currently and for upcoming events? is there any other way to hide this query apart from macro?

[splunklearner]

We are having 3 indexers with 2 cluster managers and 3 SH with one Deployer. its multi site cluster. Please help me to configure this setting before on-boarding rather than spath command? Please tell me in detail how to perform?

[splunklearner]

Hi, 

please check now

[ITWhisperer · Accepted solution]

| rex "(?<json>\{.*\})"
| spath input=json

[splunklearner]

| rex "(?<json>\{.*\})"
| spath input=json​

so the above command works fine right for mixed pattern (json and xml) for my example? currently and for upcoming events? is there any other way to hide this query apart from macro?

[splunklearner]

Thank you. It worked. One small doubt, will it be worked for upcoming new events also right? Is there any way to hide this in search rather than creating macro? 

and can we do it during on-boarding itself during index or search time extraction? Please help me

[PickleRick]

Unfortunately, for now Splunk cannot perform a structured data extraction if the whole event is not a structured data (in other words - if you have a json or XML data which has some header, like in your example, Splunk cannot automatically extract data from it).

There is an idea about it at https://ideas.splunk.com/ideas/EID-I-208 - while it's already as "Future Prospect", you can give your vote to show your support for it.

At the moment the only thing you could do would be to cut the whole header away with SEDCMD during ingestion so that all that's left is a valid json structure. But that's not always what you want.

[splunklearner]

Hi @PickleRick ,

Then what is use of KV_MODE = json that needs to be given in props.conf (saw somewhere a while ago).

Please let me understand whether my data contains both json and xml or only json? Because when i am using spath command provided by @ITWhisperer it extracted the fields... is it wrong? (if json and xml both present in my example event)

any idea on this?

[PickleRick]

I mean that KV_MODE=something works only when the _whole event_ is just a blob of structured data. Without any additional parts to it.

So KV_MODE=json will work if your whole even consists of

{"my":"data","is":"json"}

but will not work if it's

<144>2014-11-11 11:23 Some lousy[24]: pseudo-syslog header with {"json":"data","further":"down","the":street"}

[ITWhisperer]

The sample event that you posted does not contain valid json. I presume this is a copy/paste error or other typo. Please repost the raw data from your event (anonymised as required) in a code block (using the </> button above) to preserve formatting details.