topic Re: auto-decode UTF-8 encoded-string in Dashboards & Visualizations

auto-decode UTF-8 encoded-string

mushkevych — Thu, 05 Dec 2019 22:18:48 GMT

In our system we have 2 pipelines: one via Kafka->Connector->HEC->Splunk, the other DB Connect->Splunk.
Both pipelines are transporting data with the same sourcetype, which is marked with UTF-8 in props.conf: CHARSET=UTF-8.

UTF-8 strings that flow thru DB Connect are shown in Search app in their decoded format:

While the data flowing via Connector->HEC is still encoded:

Later on, the encoded values (such as "\u30a2...") are shown in the drop-down filters and so on.

Advise is very appreciated.

Re: auto-decode UTF-8 encoded-string

woodcock — Fri, 06 Dec 2019 21:11:22 GMT

Create a macro and put a giant sed command in that and then create a Calculated Field for your sourcetype. For example, here is one that I wrote to do URL decoding:

... | rex field=fieldURLencoded mode=sed "s:%25:%:g s:%20: :g s:%3C:<:g s:%3E:>:g s:%23:#:g s:%7B:{:g s:%7D:}:g s:%7C:\|:g s:%5C:\\\:g s:%5E:\^:g s:%7E:~:g s:%5B:\[:g s:%5D:\]:g s:%60:\`:g s:%3B:;:g s:%2F:/:g s:%3F:\?:g s/%3A/:/g s:%40:@:g s:%3D:=:g s:%26:&:g s:%24:\$:g s:%21:\!:g s:%2A:\*:g s:%22:\":g s:%28:\(:g s:%29:\):g s:%2B:\+:g"

Re: auto-decode UTF-8 encoded-string

mushkevych — Sat, 07 Dec 2019 00:47:04 GMT

Thank you @woodcock. it seems as manual UTF-8 decoding function. I am wondering if Splunk has anything in place for this.

Re: auto-decode UTF-8 encoded-string

woodcock — Sat, 07 Dec 2019 02:19:34 GMT

Check SplunkBase.

Re: auto-decode UTF-8 encoded-string

woodcock — Thu, 25 Jun 2020 21:33:51 GMT

My friend @Anonymous just wrote one and should be posting to splunkbase soon!

Re: auto-decode UTF-8 encoded-string

DanielP1 — Mon, 15 Jul 2024 12:08:55 GMT

Hello,

Splunk Cloud does not support "MIME Decoder Add-on for Cisco ESA". Did your colleague publish a decoder? I haven't found anything on Splunkbase.

Re: auto-decode UTF-8 encoded-string

PickleRick — Mon, 15 Jul 2024 19:26:44 GMT

Well, this is a rather old thread and Greg hasn't been much online lately. You might get bigger chance of getting a reply if you post your question about an app in a new thread. (possibly linking to this one for reference).