topic Re: Re-arranging Json fields while indexing the data in Getting Data In

Re-arranging Json fields while indexing the data

Karthikeya — Fri, 21 Feb 2025 12:35:18 GMT

This is how our normal raw event looks --

Feb 7 23:59:32 128.160.82.26 [local0.warning] <132>1 2025-02-07T23:59:32.033309Z AviVantage v-wasphictst-wdc.hc.cloud.uk.sony-443 NILVALUE NILVALUE - {"adf":true,"significant":0,"udf":false,"virtualservice":"virtualservice-e52d1117-b508-4a6d-9fb5-f03ca6319af7","report_timestamp":"2025-02-07T23:59:32.033309Z","service_engine":"GB-DRN-AB-Tier2-se-bmqhk","vcpu_id":0,"log_id":89302,"client_ip":"112.12.53.70","client_src_port":37228,"client_dest_port":443,"client_rtt":1,"request_state":"AVI_HTTP_REQUEST_STATE_SSL_HANDSHAKING","significant_log":["ADF_CLIENT_CONNECTION_CLOSED_BEFORE_REQUEST"],"vs_ip":"128.160.71.101","ocsp_status_resp_sent":true,"max_ingress_latency_fe":0,"avg_ingress_latency_fe":0,"conn_est_time_fe":1,"source_ip":"128.12.53.70","vs_name":"v-wasphictst-wdc.hc.cloud.uk.sony-443","tenant_name":"admin"}

So what we have does is removed the non-json part from this by using sedcmd and extracted the json events by giving kv_mode=json in SH. Till here it is good.

Formatted log sample -

[-]
   adf: true
   all_request_headers: { [+]
   }
   all_response_headers: { [+]
   }
   avg_ingress_latency_fe: 0
   cacheable: true
   client_dest_port: 443
   client_insights:
   client_ip: 112.11.227.250
   client_rtt: 1
   client_src_port: 34057
   compression: NO_COMPRESSION_CAN_BE_COMPRESSED
   compression_percentage: 0
   conn_est_time_fe: 1
   host: wasphictst-wdc.hc.cloud.uk.sony
   http_version: 1.1
   jwt_log: { [+]
   }
   log_id: 122364
   max_ingress_latency_fe: 0
   method: GET
   report_timestamp: 2025-02-18T16:30:29.084682Z
   request_headers: 577
   request_id: 6vT-vgq1-nSjL
   request_length: 131
   request_state: AVI_HTTP_REQUEST_STATE_READ_CLIENT_REQ_HDR
   response_code: 403
   response_content_type: text/html
   response_headers: 12
   response_length: 4181
   response_time_first_byte: 1
   response_time_last_byte: 1
   service_engine: GB-DRN-AB-Tier2-se-vxeuz
   significant: 0
   significant_log: [ [+]
   ]
   sni_hostname: wasphictst-wdc.hc.cloud.uk.sony
   source_ip: 128.11.227.250
   ssl_cipher: TLS_AES_256_GCM_SHA384
   ssl_session_id: 5032f265bd7d88f768c096bbbf78d4f2
   ssl_version: TLSv1.3
   tenant_name: admin
   udf: false
   uri_path: /cmd
   user_agent: insomnia/2021.5.3
   vcpu_id: 0
   virtualservice: virtualservice-e52d1117-b508-4a6d-9fb5-f03ca6319af7
   vs_ip: 123.160.71.101
   vs_name: v-wasphictst-wdc.hc.cloud.uk.sony-443
   waf_log: { [+]
   }
}

We want to re-arrange this fields that is we have some less information strings at the top and more info fields like (waf_log) at the bottom. how to do this re-arranging part? Checked from source end and they can't do anything from their side.

And one more thing, want waf_log to be automatically expanded by default not everytime by clicking + and again + + + in this way. Please help me in these two requirements?

Re: Re-arranging Json fields while indexing the data

ITWhisperer — Fri, 21 Feb 2025 13:44:19 GMT

What do you hope to achieve which can't be done in SPL and your dashboard searches?

Re: Re-arranging Json fields while indexing the data

livehybrid — Fri, 21 Feb 2025 14:09:17 GMT

Hi @Karthikeya

The reason waf_logs is at the bottom is because JSON events are output in alphabetical order when viewed as a JSON formatted event, and it isnt expanded because it is a child to the main event.

These are things which cannot be changed when viewing it in this way, however you could create dashboards perhaps to display the data in a table or something like that if this is preferred?

Please let me know how you get on and consider accepting this answer or adding karma this answer if it has helped.
Regards

Will

Re: Re-arranging Json fields while indexing the data

Karthikeya — Fri, 21 Feb 2025 14:13:08 GMT

Hi @ITWhisperer ,

We want the log events to be in such a manner which is useful for our app owners.

For suppose in my sample log... avg_ingress_latency_fe: 0
   cacheable: true
   client_dest_port: 443
   client_insights:

These strings which are beginning are not at all useful (but can't be removed) but waf_log which is at the bottom is more important and want this in the beginning.

@livehybrid @ITWhisperer Yes I achieved it by creating dashboard, but even after they click on any dashboard panel, they will be seeing the same less imp strings (the same event format) which is not supposed to be.

Re: Re-arranging Json fields while indexing the data

livehybrid — Fri, 21 Feb 2025 14:16:59 GMT

I see, sorry - I dont think it is possible to achieve what you are looking for without removing the fields you dont want to see from the source data.

Please let me know how you get on and consider accepting this answer or adding karma this answer if it has helped.
Regards

Will

Re: Re-arranging Json fields while indexing the data

Karthikeya — Fri, 21 Feb 2025 14:17:06 GMT

Created 4 panels for waf_logs as below:

Base Search - Index=a sourcetype=xxx:xxxx |fields * |fillnull value = "NULL"

Panel - 1

|search client_ip= "$cli_ip$" uri_query = "$uri_que$" waf_log.rule_logs{}.rule_id="$rule_id$" waf_log.rule_logs{}.rule_name="$rule_name$" waf_log.status="$log_status$" waf_log.rule_logs{}.msg="$log_mess$" |stats count by waf_log.rule_logs{}.rule_group |rename waf_log.rule_logs{}.rule_group as "Rule Group" |sort - count

Panel 2 -

|search client_ip= "$cli_ip$" uri_query = "$uri_que$" waf_log.rule_logs{}.rule_id="$rule_id$" waf_log.rule_logs{}.rule_name="$rule_name$" waf_log.status="$log_status$" waf_log.rule_logs{}.msg="$log_mess$" |stats count by waf_log.rule_logs{}.rule_id
|rename waf_log.rule_logs{}.rule_id as "Rule ID" |sort - count

Panel 3 -

|search client_ip= "$cli_ip$" uri_query = "$uri_que$" waf_log.rule_logs{}.rule_id="$rule_id$" waf_log.rule_logs{}.rule_name="$rule_name$" waf_log.status="$log_status$" waf_log.rule_logs{}.msg="$log_mess$" |stats count by waf_log.status
|rename waf_log.status as "Log Status" |sort - count

Panel 4 -

|search client_ip= "$cli_ip$" uri_query = "$uri_que$" waf_log.rule_logs{}.rule_id="$rule_id$" waf_log.rule_logs{}.rule_name="$rule_name$" waf_log.status="$log_status$" waf_log.rule_logs{}.msg="$log_mess$" |stats count by waf_log.rule_logs{}.msg
|rename waf_log.rule_logs{}.msg as "Log Message" |sort - count

Any suggestions on these dashboard to make it more readable when they click on any of the value?

Re: Re-arranging Json fields while indexing the data

ITWhisperer — Fri, 21 Feb 2025 17:03:06 GMT

Create another dashboard or panel which displays the event as you would like it and modify the drilldown on the original panel(s) to link to the new dashboard or make the new panel visible in the current dashboard.