SEDCMD usage or any alternative

Karthikeya — Wed, 19 Feb 2025 06:46:15 GMT

We have a requirement to remove few strings from the events while indexing the data. Here is my raw event sample -

{"adf":true,"significant":0,"udf":false,"virtualservice":"virtualservice-fe4a30d8-ce53-4427-b920-ec81381cb1f4","report_timestamp":"2025-02-19T06:31:56.065370Z","service_engine":"GB-DRN-AB-Tier2-se-vxeuz","vcpu_id":0,"log_id":20138,"client_ip":"128.12.73.92","client_src_port":39688,"client_dest_port":443,"client_rtt":1,"http_version":"1.1","method":"HEAD","uri_path":"/path/to/monitor/page/","host":"udg1704n01.hc.cloud.uk.sony","response_content_type":"text/html","request_length":93,"response_length":94,"response_code":400,"response_time_first_byte":1,"response_time_last_byte":1,"compression_percentage":0,"compression":"","client_insights":"","request_headers":3,"response_headers":12,"request_state":"AVI_HTTP_REQUEST_STATE_READ_CLIENT_REQ_HDR","significant_log":["ADF_HTTP_BAD_REQUEST_PLAIN_HTTP_REQUEST_SENT_ON_HTTPS_PORT","ADF_RESPONSE_CODE_4XX"],"vs_ip":"128.160.71.14","request_id":"jjc-HmSo-8zb3","max_ingress_latency_fe":0,"avg_ingress_latency_fe":0,"conn_est_time_fe":0,"source_ip":"128.12.73.92","vs_name":"v-atcptest-wdc.hc.cloud.uk.sony-443","tenant_name":"admin"}

I need to remove strings like avg_ingress_latency_fe, conn_est_time_fe, client_insights etc.

I gone through the google and found giving SEDCMD will help me. Hence giving this in props.conf and giving this in my cluster manager and it is working well.

SEDCMD-removeavglatency=s/\"avg_ingress_latency_fe\"\:[\d+]\,//g

SEDCMD-removeclientinsights=s/\"client_insights\"\:\"\.*"\,//g

But my problem we need to give more lines like this which will not be in readable format in future. I want to keep it in less lines.

Tried this but not working and in return this is disturbing the Json format-

== props.conf ==
[yourSourceType]
TRANSFORMS-removeJsonKeys = removeJsonKeys1

== transforms.conf ==
[removeJsonKeys1]
INGEST_EVAL = _raw=json_delete(_raw, "avg_ingress_latency_be", "avg_ingress_latency_fe", "max_ingress_latency_fe", "client_insights" )

because already we removed few lines from this event by giving in props.conf for auto extraction of json fields -

SEDCMD-removeheader=s/^[^\{]*//g

and here is SH props.conf -

[mysourcetype]

KV_MODE = json

AUTO_KV_JSON = true

Please suggest what can I do now instead of this to keep props.conf neat?

Re: SEDCMD usage or any alternative

bowesmana — Wed, 19 Feb 2025 21:56:01 GMT

Look at using INGEST_EVAL, where you can remove data from the JSON simply using eval statements, e.g. the following eval statement

_raw=json_delete(_raw, "avg_ingress_latency_fe", "conn_est_time_fe", "client_insights")

https://docs.splunk.com/Documentation/Splunk/9.4.0/Data/IngestEval

topic Re: SEDCMD usage or any alternative in Getting Data In

SEDCMD usage or any alternative

Re: SEDCMD usage or any alternative