UTF-8 characters - how to remove them automatically

[dabroma5]

Hi, 

I am looking for a solution to remove UTF-8 character encoding from the logs

I have a regular expression that works in the search field, but I would like to find an automated solution for Splunk Cloud.

| rex mode=sed "s/\x1B\[[0-9;]*[mK]//g"

Sample log line:

2023-11-15 11:47:21,605 backend_2023.2.8: INFO  [-dispatcher-7] vip.service.northbound.MrpServiceakkaAddress=akka://backend, akkaUid=2193530468036521242 MRP Service is alive and active.

Any idea?

Thanks for help. 

[PickleRick]

1. UTF-8 includes normal ASCII range. I don't think that's what you meant by "remove UTF-8 characters". UTF-8 is just an encoding.

2. What you're presenting are so called ANSI escape sequences.

3. Are you sure they are literarily in your logs or do you have them rendered and filtered already?

4. Ugh. Where are you getting those events from? It seems like capturing some terminal input instead of sending events as such. (BTW, you could try setting some dumb terminal type before starting your process so the service doesn't produce such ugly codes).

[dabroma5]

This is what it looks like straight from the log file:

2023-11-15 11:47:21,605 backend_2023.2.8: INFO  [-dispatcher-7] vip.service.northbound.MrpService.serverakkaAddress=akka://backend, akkaUid=2193530468036521242 Server is alive - num conns = 0

of course it looks better from the terminal

[PickleRick]

To be honest, I'm not fully sure at which step of the pipeline (if any) those non-printable characters are escaped. I'll have to verify it.

But still - it would be best if you could make the source generate logs without the formating codes - they don't belong there. It's a presentation layer, those codes shouldn't be in the log entries.

[dabroma5]

Hi

I have a log file which contains UTF-8 characters

"[1;33mWARN  [-dispatcher-6] " and so on.

Below regex works perfectly, but how to automate this solution

| rex mode=sed "s/\x1B\[[0-9;]*[mK]//g"

Thanks for your help.

[dabroma5]

which one should I move to /opt/splunkforwarder/etc/system/local , and edit:

/opt/splunkforwarder/etc/system/default/props.conf
/opt/splunkforwarder/etc/apps/search/default/props.conf
/opt/splunkforwarder/etc/apps/splunk_internal_metrics/default/props.conf
/opt/splunkforwarder/etc/apps/learned/local/props.conf
/opt/splunkforwarder/etc/apps/SplunkUniversalForwarder/default/props.conf
/opt/splunkforwarder/var/run/splunk/confsnapshot/baseline_local/apps/learned/local/props.conf
/opt/splunkforwarder/var/run/splunk/confsnapshot/baseline_default/system/default/props.conf
/opt/splunkforwarder/var/run/splunk/confsnapshot/baseline_default/apps/search/default/props.conf
/opt/splunkforwarder/var/run/splunk/confsnapshot/baseline_default/apps/splunk_internal_metrics/default/props.conf
/opt/splunkforwarder/var/run/splunk/confsnapshot/baseline_default/apps/SplunkUniversalForwarder/default/props.conf

[richgalloway]

None of those.  The SEDCMD setting must be on the indexer(s) and/or heavy forwarders.  It should go in the stanza where the sourcetype it goes with resides (if the file is in a default stanza then put the setting in the associated local directory).

[dabroma5]

I can't make it work.

I found some explanation here:

https://community.splunk.com/t5/Getting-Data-In/How-to-replace-characters-in-logs-using-SEDCMD-in-pr...

but they said the change should be made in HF props.conf

I need to make it work on UF for Splunk Cloud

[richgalloway]

Use that regex in SEDCMD in props.conf.

[mysourcetype]
SEDCMD-no_UTF-8 = s/\x1B\[[0-9;]*[mK]//g

[dabroma5]

Unfortunately, this is not an option for Splunk Cloud

[richgalloway]

Splunk Cloud fully supports SEDCMD.