Exclude or Remove few fields while on-boarding data

[Karthikeya]

We have a requirement to exclude or remove few fields from the event we receive it in Splunk. Already we have extracted json data by giving condition in props.conf and below is the sample event -

{ [-]
   adf: true
   all_request_headers: { [+]
   }
   all_response_headers: { [+]
   }
   avg_ingress_latency_be: 0
   avg_ingress_latency_fe: 0
   cacheable: true
   client_dest_port: 443
   client_insights:
   client_ip: XXXXXXXX
   client_rtt: 1
   client_src_port: 13353
   compression: NO_COMPRESSION_CAN_BE_COMPRESSED
   compression_percentage: 0
   conn_est_time_be: 6
   conn_est_time_fe: 0
   headers_received_from_server: { [+]
   }
   headers_sent_to_server: { [+]
   }
   host: wasphictst-wdc.hc.cloud.uk.sony
   http_version: 1.1
   jwt_log: { [+]
   }
   log_id: 121721
   max_ingress_latency_be: 0
   max_ingress_latency_fe: 0
   method: GET
   persistent_session_id: 3472328296699025517
   pool: pool-cac2726e-acd1-4225-8ac8-72ebd82a57a6
   pool_name: p-wasphictst-wdc.hc.cloud.uk.sony-wdc-443
   report_timestamp: 2025-02-18T11:33:23.069736Z
   request_headers: 577
   request_id: euh-xfiN-7Ikq
   request_length: 148
   request_state: AVI_HTTP_REQUEST_STATE_SEND_RESPONSE_BODY_TO_CLIENT
   response_code: 404
   response_content_type: text/html; charset=iso-8859-1
   response_headers: 13
   response_length: 6148
   response_time_first_byte: 61
   response_time_last_byte: 61
   rewritten_uri_query: test=%26%26%20whoami
   server_conn_src_ip: 128.160.77.237
   server_dest_port: 80
   server_ip: 128.160.73.123
   server_name: 128.160.73.123
   server_response_code: 404
   server_response_length: 373
   server_response_time_first_byte: 52
   server_response_time_last_byte: 61
   server_rtt: 9
   server_src_port: 56233
   servers_tried: 1
   service_engine: GB-DRN-AB-Tier2-se-vxeuz
   significant: 0
   significant_log: [ [+]
   ]
   sni_hostname: wasphictst-wdc.hc.cloud.uk.sony
   source_ip: 128.164.6.186
   ssl_cipher: TLS_AES_256_GCM_SHA384
   ssl_session_id: 935810081909dc8c018416502ece5d00
   ssl_version: TLSv1.3
   tenant_name: admin
   udf: false
   uri_path: /cmd
   uri_query: test=&& whoami
   user_agent: insomnia/2021.5.3
   vcpu_id: 0
   virtualservice: virtualservice-e52d1117-b508-4a6d-9fb5-f03ca6319af7
   vs_ip: 128.160.71.101
   vs_name: v-wasphictst-wdc.hc.cloud.uk.sony-443
   waf_log: { [+]
   }
}

We need to remove few fields from new and existing events like "avg_ingress_latency_be",  "avg_ingress_latency_fe", "request_state", "server_response_code" and many of the fields while onboarding. Where can I write the logic to exclude these fields because user app owners don't want these fields while viewing the data and source cannot edit that. We need to do this before on-boarding.

[Karthikeya]

Given this in props.conf 

SEDCMD-removeevents= s/\"avg_ingress_latency_fe\":.*//g    as per raw data but it is not doing anything in return it is disturbing the json format what we have given in SH.

SH props.conf

[mysourcetype]

KV_MODE = json

AUTO_KV_JSON = true

Please help me in this case....

[Karthikeya]

Here is the raw data sample-- 

{"adf":true,"significant":0,"udf":false,"virtualservice":"virtualservice-fe4a30d8-ce53-4427-b920-ec81381cb1f4","report_timestamp":"2025-02-18T17:21:53.173205Z","service_engine":"GB-DRN-AB-Tier2-se-vxeuz","vcpu_id":0,"log_id":18544,"client_ip":"128.12.73.92","client_src_port":42996,"client_dest_port":443,"client_rtt":1,"http_version":"1.1","method":"HEAD","uri_path":"/path/to/monitor/page/","host":"udg1704n01.hc.cloud.uk.sony","response_content_type":"text/html","request_length":93,"response_length":94,"response_code":400,"response_time_first_byte":1,"response_time_last_byte":1,"compression_percentage":0,"compression":"","client_insights":"","request_headers":3,"response_headers":12,"request_state":"AVI_HTTP_REQUEST_STATE_READ_CLIENT_REQ_HDR","significant_log":["ADF_HTTP_BAD_REQUEST_PLAIN_HTTP_REQUEST_SENT_ON_HTTPS_PORT","ADF_RESPONSE_CODE_4XX"],"vs_ip":"128.160.71.14","request_id":"2OP-U2vt-pre1","max_ingress_latency_fe":0,"avg_ingress_latency_fe":0,"conn_est_time_fe":1,"source_ip":"128.12.73.92","vs_name":"v-atcptest-wdc.hc.cloud.uk.hc-443","tenant_name":"admin"}

[livehybrid]

Hi @Karthikeya 

This should be really easy to achieve by adding some simple props/transforms to your Indexers or HFs:

== props.conf ==
[yourSourceType]
TRANSFORMS-removeJsonKeys = removeJsonKeys1

== transforms.conf ==
[removeJsonKeys1]
INGEST_EVAL = _raw=json_delete(_raw, "key1", "nestedkey.subkey2")

You can also see how this would work in the UI, although obviously this isnt persistent.  Here is an example working to see:

SPL

| makeresults 
|  eval _raw = "[{\"integrationName\":\"Opsgenie Edge Connector - Splunk\",\"alert\":{\"message\":\"[ThousandEyes] Alert for TMS Core Healthcheck\",\"id\":\"abc123xyz\"},\"action\":\"Create\"},{\"integrationName\":\"Opsgenie Edge Connector - Splunk\",\"alert\":{\"message\":\"[ThousandEyes] Alert for TMS Core Healthcheck\",\"id\":\"abc123xyz\"},\"action\":\"Close\"},{\"integrationName\":\"Opsgenie Edge Connector - Splunk\",\"alert\":{\"message\":\"[ThousandEyes] Alert for TMS Core Healthcheck\",\"id\":\"def456uvw\"},\"action\":\"Create\"},{\"integrationName\":\"Opsgenie Edge Connector - Splunk\",\"alert\":{\"message\":\"[ThousandEyes] Alert for TMS Core Healthcheck\",\"id\":\"def456uvw\"},\"action\":\"Close\"},{\"integrationName\":\"Opsgenie Edge Connector - Splunk\",\"alert\":{\"message\":\"[ThousandEyes] Alert for TMS Core Healthcheck\",\"id\":\"ghi789rst\"},\"action\":\"Create\"}]"
| eval events=json_array_to_mv(_raw)
| mvexpand events
| eval _raw=events
| fields _raw
| eval _raw=json_delete(_raw, "integrationName", "alert.id")

Please let me know how you get on and consider accepting this answer or adding karma this answer if it has helped.
Regards

Will

[livehybrid]

So you would use

== props.conf ==
[yourSourceType]
TRANSFORMS-removeJsonKeys = removeJsonKeys1

== transforms.conf ==
[removeJsonKeys1]
INGEST_EVAL = _raw=json_delete(_raw, "avg_ingress_latency_be", "avg_ingress_latency_fe", "request_state", "server_response_code" )

as json_delete takes an object (_raw) and a list of keys to delete.

Please let me know how you get on and consider accepting this answer or adding karma this answer if it has helped.
Regards

Will

[Karthikeya]

Thanks @livehybrid for the reply. However I have given the same but it is not working as expected. As I earlier said this is not by default JSON data we converted it by using KV_MODE = json in SH... I think JSON is extracting at search time but I have given this in index time. That might be the reason this json_delete not working... can you please help me with any other alternative?

[livehybrid]

Hi @Karthikeya 

Is the sample data you provided after you have modified it with any SPL, or is that as it arrives into Splunk?

It looks like its already a JSON string when it arrives, if so then json functions should work. I will test this further.

[Karthikeya]

is it not complete JSON when it arrives. Its a raw data which I have removed unwanted lines in master props.conf by giving SEDCMD and then wrote kv_mode in SH.

[Karthikeya]

Anyone please help in this