Strip control/color codes

[oreoshake]

I have log files with color codes and control characters that we'd like to strip because they clutter the search results. In a few cases, we've just removed them from the file with the commands found at http://www.commandlinefu.com/commands/view/3584/remove-color-codes-special-characters-with-sed

It used to be a one off case, but now it is the standard case. It seems there is some global setting I might be missing. Or is this something that I would have to use a transform for?

EDIT

So based on http://www.splunk.com/base/Documentation/4.1.4/Admin/Configurecharactersetencoding
which says "Splunk escapes the invalid characters as hex values (for example: "\xF3")."

It appears my file encoding was set wrong. Is there a way to just tell splunk to ignore the characters?

[Ayn · Accepted solution]

You can use the SEDCMD directive in props.conf for this.

For instance, let's call the sourcetype you want this to apply to "colorlogs". In props.conf, specify this:

[colorlogs]
SEDCMD-removecolorcodes = s/\x1B\[([0-9]{1,2}(;[0-9]{1,2})?)?[m|K]//g

After restarting Splunk, all color codes should be removed from any events of sourcetype "colorlogs" added to the index in the future.

(note: the regex itself is just the same as the one in the link you supplied, I'm taking their word for that it's actually effective in removing color codes 😉 )

[sogeniusio]

Huge thumbs up, been at this all day!! Thanks!

[mattymo]

greetings from 2020. still working to this day. Now relevant with kubernetes logging.

thanks for all the wisdom over the years @Ayn

[Ayn]

Great! 🙂 Could you please mark the answer as accepted? That way others see that the question has an accepted solution.

[xplodersuv]

Actually nevermind, it works!

[xplodersuv]

Thanks Ayn, makes sense. I put this in apps/search/local/props.conf but the existing data isn't clean. Is that an index or search time deal?

Am I missing a transform?

[cssmdi]

Hi
The SEDCMD is done at index time bevore the events are stored. The stored events have the color codes stripped off.

Maybe there is also a problem with the order of commands. When I changed the sourcetype and deleted the color codes, I had to put the SEDCMD in props.conf before the REPORT... to change the sourcetype.

It would be helpful to have the possibility to remove color codes included directly in splunk.