We have a requirement to remove few strings from the events while indexing the data. Here is my raw event sample -
{"adf":true,"significant":0,"udf":false,"virtualservice":"virtualservice-fe4a30d8-ce53-4427-b920-ec81381cb1f4","report_timestamp":"2025-02-19T06:31:56.065370Z","service_engine":"GB-DRN-AB-Tier2-se-vxeuz","vcpu_id":0,"log_id":20138,"client_ip":"128.12.73.92","client_src_port":39688,"client_dest_port":443,"client_rtt":1,"http_version":"1.1","method":"HEAD","uri_path":"/path/to/monitor/page/","host":"udg1704n01.hc.cloud.uk.sony","response_content_type":"text/html","request_length":93,"response_length":94,"response_code":400,"response_time_first_byte":1,"response_time_last_byte":1,"compression_percentage":0,"compression":"","client_insights":"","request_headers":3,"response_headers":12,"request_state":"AVI_HTTP_REQUEST_STATE_READ_CLIENT_REQ_HDR","significant_log":["ADF_HTTP_BAD_REQUEST_PLAIN_HTTP_REQUEST_SENT_ON_HTTPS_PORT","ADF_RESPONSE_CODE_4XX"],"vs_ip":"128.160.71.14","request_id":"jjc-HmSo-8zb3","max_ingress_latency_fe":0,"avg_ingress_latency_fe":0,"conn_est_time_fe":0,"source_ip":"128.12.73.92","vs_name":"v-atcptest-wdc.hc.cloud.uk.sony-443","tenant_name":"admin"}
I need to remove strings like avg_ingress_latency_fe, conn_est_time_fe, client_insights etc.
I gone through the google and found giving SEDCMD will help me. Hence giving this in props.conf and giving this in my cluster manager and it is working well.
SEDCMD-removeavglatency=s/\"avg_ingress_latency_fe\"\:[\d+]\,//g
SEDCMD-removeclientinsights=s/\"client_insights\"\:\"\.*"\,//g
But my problem we need to give more lines like this which will not be in readable format in future. I want to keep it in less lines.
Tried this but not working and in return this is disturbing the Json format-
== props.conf ==
[yourSourceType]
TRANSFORMS-removeJsonKeys = removeJsonKeys1
== transforms.conf ==
[removeJsonKeys1]
INGEST_EVAL = _raw=json_delete(_raw, "avg_ingress_latency_be", "avg_ingress_latency_fe", "max_ingress_latency_fe", "client_insights" )
because already we removed few lines from this event by giving in props.conf for auto extraction of json fields -
SEDCMD-removeheader=s/^[^\{]*//g and here is SH props.conf - [mysourcetype]KV_MODE = jsonAUTO_KV_JSON = true Please suggest what can I do now instead of this to keep props.conf neat?
