I have the following rex substitution in a query to aggregate various log messages (with the string Liveness and Readiness):
index=k8s ("event.go") AND (kind="Pod") AND (type="Warning" OR type="Error") source="*kubelet.log" | rex mode=sed "s/(object=\"[^\s]+\")(.*)Liveness(.*)/\1 message=\"Liveness Error\"/g" | rex mode=sed "s/(object=\"[^\s]+\")(.*)Readiness(.*)/\1 message=\"Readiness Error\"/g"| dedup object message
The above appears to work correctly and provide the desired result.
For example, the above transforms events like below:
I0903 17:12:49.308289 2024433 event.go:211] "Event occurred" object="namespace1/podfoo" message="Readiness probe failed: + cd /sandbox\\n++ curl --output /dev/null --max-time 28 --silent --write-out '%{http_code}' http://0.0.0.0:20012/heartbeat\\n+ ret=000\\n+ for expected_status in 200\\n+ [[ 000 == 200 ]]\\n+ [[ '' == \\\\t\\\\r\\\\u\\\\e ]]\\n+ false\\n"
nicely into the following:
I0903 17:12:49.308289 2024433 event.go:211] "Event occurred" object="namespace1/podfoo" message="Readiness Error"
However when I try to stream the above query into stats ("stats count by message"), the transformed events generated as part of the rex substitution disappears for some reason and stats seem to be acting on the original event messages (as if the rex sed had no effect).
index=k8s ("event.go") AND (kind="Pod") AND (type="Warning" OR type="Error") source="*kubelet.log" | rex mode=sed "s/(object=\"[^\s]+\")(.*)Liveness(.*)/\1 message=\"Liveness Error\"/g" | rex mode=sed "s/(object=\"[^\s]+\")(.*)Readiness(.*)/\1 message=\"Readiness Error\"/g"| dedup object message | stats count by message
With the above, stats appears to aggregate on the original message contents of the events rather than the output of the rex substitution. For example, I see:
message count
Readiness probe errored: rpc error: code = Unknown ... 1059
Readiness probe failed: HTTP probe failed with statuscode: 503 2003
rather than the substituted message fields aggregated to something along the lines of
message count
Readiness Error 3062
How can I get the output of the rex sed (like in the example above) to pass the substituted message fields in the events to stats?
