topic Re: JSON format - EVAL-_raw = gives duplicate content in Getting Data In

JSON format - EVAL-_raw = gives duplicate content

petreb — Fri, 28 Feb 2020 16:46:40 GMT

Hello,

I am trying to display at search time only the content of the "log" field - where the application data is.

I am using the stanza below on the SH

cheers,

[source::http:k8s_test]
KV_MODE = json
EVAL-_raw = log

_raw event

 { [-]
   K8Cluster: k8s-cluster-aa-bb-01
   docker: { [-]
     container_id: 919d689b4ee5aa0ac2ad7ac3333557b4bb7471da313ac9c7e6cbfc9c9e925e8a
   }
   kubernetes: { [+]
   }
   log: [2020/02/28 16:40:41] [error] [out_fw] no upstream connections available
   stream: stderr
}

output

[2020/02/28 16:30:18] [error] [out_fw] no upstream connections available 
[2020/02/28 16:30:18] [error] [out_fw] no upstream connections available

Re: JSON format - EVAL-_raw = gives duplicate content

masonmorales — Fri, 28 Feb 2020 18:24:05 GMT

Why not just do a | table fields _time log?
If you look at the raw events in search and expand a single event using the ">" under the "i" column (next to Time), do you see duplicate values for each of the fields? If so, that's a different problem, usually caused by both INDEXED_FIELDS = json enabled on the HF/indexer as well as KV_MODE = json being configured on the search head, which results in duplicate values in the JSON field extractions.

Re: JSON format - EVAL-_raw = gives duplicate content

petreb — Wed, 30 Sep 2020 04:22:42 GMT

I would like to do this transparently for the user, without piped commands

the events come via HEC, and when I remove the EVAL-_raw = log line everything is displayed correctly in json format; I want to display only the content of the "log" field

INDEXED_EXTRACTIONS is not set, and removing KV_MODE on the SH doesn't get rid of the additional line

looks like EVAL doubles the values of all fields (excepting the default ones), not only log

Re: JSON format - EVAL-_raw = gives duplicate content

to4kawa — Fri, 28 Feb 2020 19:09:35 GMT

_raw is default indexed event field.

EVAL - _raw = log

does not replace indexed event.

| makeresults 
| eval _raw="{
    K8Cluster: k8s-cluster-aa-bb-01
    docker: { [-]
      container_id: 919d689b4ee5aa0ac2ad7ac3333557b4bb7471da313ac9c7e6cbfc9c9e925e8a
    }
    kubernetes: { [+]
    }
    log: [2020/02/28 16:40:41] [error] [out_fw] no upstream connections available
    stream: stderr
 } " 
| rex mode=sed "s/(?s).*(log\:.+?) stream.*/\1/"

As this result,
props.conf

SEDCMD - log = s/(?s).*(log\:.+?) stream.*/\1/

but this depends on your LINE_BREAKER.

Re: JSON format - EVAL-_raw = gives duplicate content

petreb — Fri, 28 Feb 2020 20:42:22 GMT

following your example I have tried with the format below, but somehow it interprets the string between square brackets as splunk command

| makeresults 
     | eval _raw="{"log":"[2020/02/28 18:38:00] [error] [out_fw] no upstream connections available","stream":"stderr","docker":{"container_id":"736f7b10a0bda6b97267d8f51e9a8c1fbf8f8f41edea0f3d79b174b7dc5f48cb"},"kubernetes":{"container_name":"billing","namespace_name":"pks-system","pod_name":"telemetry-agent-77f797c749-qj9rv","container_image":"pkstelemetrybot/telemetry-agent:latest","container_image_id":"docker://sha256:c9dbff3df8b19ee2b91df08982cbdb0047bd8e0d830acba73e03959db80c6928","pod_id":"dbbeee1f-63a4-4b03-a74b-0a53b5db44e8","labels":{"app":"telemetry-agent","pod-template-hash":"77f797c749"},"host":"77141ae5-920f-4f65-bd0a-ba78a85c157d","master_url":"https://10.111.192.1:443/api","namespace_id":"6e78cc1d-b345-44c3-b8e3-5c1af076afc5"},"K8Cluster":"k8s-cluster-test-kr-01"}"
     | rex mode=sed "s/\{("log"\:\".+?)\","stream.*/\1/"

Re: JSON format - EVAL-_raw = gives duplicate content

to4kawa — Fri, 28 Feb 2020 22:59:25 GMT

| makeresults 
| eval _raw="{\"log\":\"[2020/02/28 18:38:00] [error] [out_fw] no upstream connections available\",\"stream\":\"stderr\",\"docker\":{\"container_id\":\"736f7b10a0bda6b97267d8f51e9a8c1fbf8f8f41edea0f3d79b174b7dc5f48cb\"},\"kubernetes\":{\"container_name\":\"billing\",\"namespace_name\":\"pks-system\",\"pod_name\":\"telemetry-agent-77f797c749-qj9rv\",\"container_image\":\"pkstelemetrybot/telemetry-agent:latest\",\"container_image_id\":\"docker://sha256:c9dbff3df8b19ee2b91df08982cbdb0047bd8e0d830acba73e03959db80c6928\",\"pod_id\":\"dbbeee1f-63a4-4b03-a74b-0a53b5db44e8\",\"labels\":{\"app\":\"telemetry-agent\",\"pod-template-hash\":\"77f797c749\"},\"host\":\"77141ae5-920f-4f65-bd0a-ba78a85c157d\",\"master_url\":\"https://10.111.192.1:443/api\",\"namespace_id\":\"6e78cc1d-b345-44c3-b8e3-5c1af076afc5\"},\"K8Cluster\":\"k8s-cluster-test-kr-01\"}" 
| rex mode=sed "s/\{(\"log\"\:\".+?),.*/\1/"

JSON has many "". press ⌘(OR Ctrl)+F *2 and convert " to \"

Re: JSON format - EVAL-_raw = gives duplicate content

petreb — Mon, 09 Mar 2020 15:16:38 GMT

i did manage to fix it with

EVAL-_raw = replace(_raw, "^\{\"log\"\:\"(.+?)\",\"stream.*", "\1")

looks like SEDCMD works only at index time, I was operating at search time level

SEDCMD- =
* Only used at index time.