topic Re: Strip control/color codes in Getting Data In

Strip control/color codes

oreoshake — Sun, 24 Jul 2011 22:14:16 GMT

I have log files with color codes and control characters that we'd like to strip because they clutter the search results. In a few cases, we've just removed them from the file with the commands found at http://www.commandlinefu.com/commands/view/3584/remove-color-codes-special-characters-with-sed

It used to be a one off case, but now it is the standard case. It seems there is some global setting I might be missing. Or is this something that I would have to use a transform for?

EDIT

So based on http://www.splunk.com/base/Documentation/4.1.4/Admin/Configurecharactersetencoding
which says "Splunk escapes the invalid characters as hex values (for example: "\xF3")."

It appears my file encoding was set wrong. Is there a way to just tell splunk to ignore the characters?

Re: Strip control/color codes

Ayn — Mon, 25 Jul 2011 06:28:00 GMT

You can use the SEDCMD directive in props.conf for this.

For instance, let's call the sourcetype you want this to apply to "colorlogs". In props.conf, specify this:

[colorlogs]
SEDCMD-removecolorcodes = s/\x1B\[([0-9]{1,2}(;[0-9]{1,2})?)?[m|K]//g

After restarting Splunk, all color codes should be removed from any events of sourcetype "colorlogs" added to the index in the future.

(note: the regex itself is just the same as the one in the link you supplied, I'm taking their word for that it's actually effective in removing color codes 😉 )

Re: Strip control/color codes

xplodersuv — Tue, 26 Jul 2011 19:10:23 GMT

Thanks Ayn, makes sense. I put this in apps/search/local/props.conf but the existing data isn't clean. Is that an index or search time deal?

Am I missing a transform?

Re: Strip control/color codes

xplodersuv — Tue, 26 Jul 2011 19:17:03 GMT

Actually nevermind, it works!

Re: Strip control/color codes

Ayn — Tue, 26 Jul 2011 21:19:12 GMT

Great! 🙂 Could you please mark the answer as accepted? That way others see that the question has an accepted solution.

Re: Strip control/color codes

sogeniusio — Sat, 21 Oct 2017 22:06:58 GMT

Huge thumbs up, been at this all day!! Thanks!

Re: Strip control/color codes

mattymo — Wed, 29 Jan 2020 15:53:43 GMT

greetings from 2020. still working to this day. Now relevant with kubernetes logging.

thanks for all the wisdom over the years @Ayn

Re: Strip control/color codes

cssmdi — Thu, 10 Jun 2021 16:16:14 GMT

Hi
The SEDCMD is done at index time bevore the events are stored. The stored events have the color codes stripped off.

Maybe there is also a problem with the order of commands. When I changed the sourcetype and deleted the color codes, I had to put the SEDCMD in props.conf before the REPORT... to change the sourcetype.

It would be helpful to have the possibility to remove color codes included directly in splunk.

Re: Strip control/color codes

vchepkov — Thu, 25 Jul 2024 17:48:55 GMT

I know it's an old post, but it helped me , but it leaves `[0;m` behind, which is 'Reset' I believe

Re: Strip control/color codes

vchepkov — Thu, 25 Jul 2024 18:05:41 GMT

I just added additional

SEDCMD-removereset=s/\x1B\[0;m//g