auto-decode UTF-8 encoded-string

[mushkevych]

In our system we have 2 pipelines: one via Kafka->Connector->HEC->Splunk, the other DB Connect->Splunk.
Both pipelines are transporting data with the same sourcetype, which is marked with UTF-8 in props.conf: CHARSET=UTF-8.

UTF-8 strings that flow thru DB Connect are shown in Search app in their decoded format:

While the data flowing via Connector->HEC is still encoded:

Later on, the encoded values (such as "\u30a2...") are shown in the drop-down filters and so on.

Advise is very appreciated.

[woodcock]

Create a macro and put a giant sed command in that and then create a Calculated Field for your sourcetype. For example, here is one that I wrote to do URL decoding:

... | rex field=fieldURLencoded mode=sed "s:%25:%:g s:%20: :g s:%3C:<:g s:%3E:>:g s:%23:#:g s:%7B:{:g s:%7D:}:g s:%7C:\|:g s:%5C:\\\:g s:%5E:\^:g s:%7E:~:g s:%5B:\[:g s:%5D:\]:g s:%60:\`:g s:%3B:;:g s:%2F:/:g s:%3F:\?:g s/%3A/:/g s:%40:@:g s:%3D:=:g s:%26:&:g s:%24:\$:g s:%21:\!:g s:%2A:\*:g s:%22:\":g s:%28:\(:g s:%29:\):g s:%2B:\+:g"

[mushkevych]

Thank you @woodcock. it seems as manual UTF-8 decoding function. I am wondering if Splunk has anything in place for this.

[woodcock]

My friend @trmoser just wrote one and should be posting to splunkbase soon!

[woodcock]

Check SplunkBase.