topic UTF-8 characters - how to remove them automatically in Getting Data In

UTF-8 characters - how to remove them automatically

dabroma5 — Thu, 16 Nov 2023 13:06:20 GMT

Hi,

I am looking for a solution to remove UTF-8 character encoding from the logs

I have a regular expression that works in the search field, but I would like to find an automated solution for Splunk Cloud.

| rex mode=sed "s/\x1B\[[0-9;]*[mK]//g"

Sample log line:

2023-11-15 11:47:21,605 backend_2023.2.8: [36mINFO [0;39m [-dispatcher-7] [36mvip.service.northbound.MrpServiceakkaAddress=akka://backend, akkaUid=2193530468036521242[0;39m [39mMRP Service is alive and active.[0;39m

Any idea?

Thanks for help.

[Splunk Cloud] UTF-8 characters in a log file - how to remove it automatically

dabroma5 — Wed, 15 Nov 2023 15:07:25 GMT

I have a log file which contains UTF-8 characters

"[1;33mWARN [0;39m [-dispatcher-6] [36m" and so on.

Below regex works perfectly, but how to automate this solution

| rex mode=sed "s/\x1B\[[0-9;]*[mK]//g"

Thanks for your help.

Re: UTF-8 characters - how to remove them automatically

PickleRick — Thu, 16 Nov 2023 13:30:02 GMT

1. UTF-8 includes normal ASCII range. I don't think that's what you meant by "remove UTF-8 characters". UTF-8 is just an encoding.

2. What you're presenting are so called ANSI escape sequences.

3. Are you sure they are literarily in your logs or do you have them rendered and filtered already?

4. Ugh. Where are you getting those events from? It seems like capturing some terminal input instead of sending events as such. (BTW, you could try setting some dumb terminal type before starting your process so the service doesn't produce such ugly codes).

Re: [Splunk Cloud] UTF-8 characters in a log file - how to remove it automatically

richgalloway — Thu, 16 Nov 2023 15:54:27 GMT

Use that regex in SEDCMD in props.conf.

[mysourcetype] SEDCMD-no_UTF-8 = s/\x1B\[[0-9;]*[mK]//g

Re: [Splunk Cloud] UTF-8 characters in a log file - how to remove it automatically

dabroma5 — Fri, 17 Nov 2023 08:47:07 GMT

Unfortunately, this is not an option for Splunk Cloud

Re: UTF-8 characters - how to remove them automatically

dabroma5 — Fri, 17 Nov 2023 08:57:24 GMT

This is what it looks like straight from the log file:

2023-11-15 11:47:21,605 backend_2023.2.8: [36mINFO [0;39m [-dispatcher-7] [36mvip.service.northbound.MrpService.serverakkaAddress=akka://backend, akkaUid=2193530468036521242[0;39m [39mServer is alive - num conns = 0[0;39m

of course it looks better from the terminal

Re: UTF-8 characters - how to remove them automatically

PickleRick — Fri, 17 Nov 2023 09:57:31 GMT

To be honest, I'm not fully sure at which step of the pipeline (if any) those non-printable characters are escaped. I'll have to verify it.

But still - it would be best if you could make the source generate logs without the formating codes - they don't belong there. It's a presentation layer, those codes shouldn't be in the log entries.

Re: [Splunk Cloud] UTF-8 characters in a log file - how to remove it automatically

richgalloway — Fri, 17 Nov 2023 13:23:47 GMT

Splunk Cloud fully supports SEDCMD.

Re: [Splunk Cloud] UTF-8 characters in a log file - how to remove it automatically

dabroma5 — Fri, 17 Nov 2023 14:03:49 GMT

which one should I move to /opt/splunkforwarder/etc/system/local , and edit:

/opt/splunkforwarder/etc/system/default/props.conf
/opt/splunkforwarder/etc/apps/search/default/props.conf
/opt/splunkforwarder/etc/apps/splunk_internal_metrics/default/props.conf
/opt/splunkforwarder/etc/apps/learned/local/props.conf
/opt/splunkforwarder/etc/apps/SplunkUniversalForwarder/default/props.conf
/opt/splunkforwarder/var/run/splunk/confsnapshot/baseline_local/apps/learned/local/props.conf
/opt/splunkforwarder/var/run/splunk/confsnapshot/baseline_default/system/default/props.conf
/opt/splunkforwarder/var/run/splunk/confsnapshot/baseline_default/apps/search/default/props.conf
/opt/splunkforwarder/var/run/splunk/confsnapshot/baseline_default/apps/splunk_internal_metrics/default/props.conf
/opt/splunkforwarder/var/run/splunk/confsnapshot/baseline_default/apps/SplunkUniversalForwarder/default/props.conf

Re: [Splunk Cloud] UTF-8 characters in a log file - how to remove it automatically

richgalloway — Fri, 17 Nov 2023 15:39:59 GMT

None of those. The SEDCMD setting must be on the indexer(s) and/or heavy forwarders. It should go in the stanza where the sourcetype it goes with resides (if the file is in a default stanza then put the setting in the associated local directory).

Re: [Splunk Cloud] UTF-8 characters in a log file - how to remove it automatically

dabroma5 — Mon, 20 Nov 2023 08:31:07 GMT

I can't make it work.

I found some explanation here:

https://community.splunk.com/t5/Getting-Data-In/How-to-replace-characters-in-logs-using-SEDCMD-in-props-conf-in/m-p/392306

but they said the change should be made in HF props.conf

I need to make it work on UF for Splunk Cloud