Getting Data In

Strip control/color codes

oreoshake
Communicator

I have log files with color codes and control characters that we'd like to strip because they clutter the search results. In a few cases, we've just removed them from the file with the commands found at http://www.commandlinefu.com/commands/view/3584/remove-color-codes-special-characters-with-sed

It used to be a one off case, but now it is the standard case. It seems there is some global setting I might be missing. Or is this something that I would have to use a transform for?

EDIT

So based on http://www.splunk.com/base/Documentation/4.1.4/Admin/Configurecharactersetencoding
which says "Splunk escapes the invalid characters as hex values (for example: "\xF3")."

It appears my file encoding was set wrong. Is there a way to just tell splunk to ignore the characters?

Tags (1)
1 Solution

Ayn
Legend

You can use the SEDCMD directive in props.conf for this.

For instance, let's call the sourcetype you want this to apply to "colorlogs". In props.conf, specify this:

[colorlogs]
SEDCMD-removecolorcodes = s/\x1B\[([0-9]{1,2}(;[0-9]{1,2})?)?[m|K]//g

After restarting Splunk, all color codes should be removed from any events of sourcetype "colorlogs" added to the index in the future.

(note: the regex itself is just the same as the one in the link you supplied, I'm taking their word for that it's actually effective in removing color codes 😉 )

View solution in original post

Ayn
Legend

You can use the SEDCMD directive in props.conf for this.

For instance, let's call the sourcetype you want this to apply to "colorlogs". In props.conf, specify this:

[colorlogs]
SEDCMD-removecolorcodes = s/\x1B\[([0-9]{1,2}(;[0-9]{1,2})?)?[m|K]//g

After restarting Splunk, all color codes should be removed from any events of sourcetype "colorlogs" added to the index in the future.

(note: the regex itself is just the same as the one in the link you supplied, I'm taking their word for that it's actually effective in removing color codes 😉 )

sogeniusio
Path Finder

Huge thumbs up, been at this all day!! Thanks!

0 Karma

mattymo
Splunk Employee
Splunk Employee

greetings from 2020. still working to this day. Now relevant with kubernetes logging.

thanks for all the wisdom over the years @Ayn

- MattyMo
0 Karma

Ayn
Legend

Great! 🙂 Could you please mark the answer as accepted? That way others see that the question has an accepted solution.

0 Karma

xplodersuv
New Member

Actually nevermind, it works!

0 Karma

xplodersuv
New Member

Thanks Ayn, makes sense. I put this in apps/search/local/props.conf but the existing data isn't clean. Is that an index or search time deal?

Am I missing a transform?

0 Karma

cssmdi
Explorer

Hi
The SEDCMD is done at index time bevore the events are stored. The stored events have the color codes stripped off.

Maybe there is also a problem with the order of commands. When I changed the sourcetype and deleted the color codes, I had to put the SEDCMD in props.conf before the REPORT... to change the sourcetype.

It would be helpful to have the possibility to remove color codes included directly in splunk.

0 Karma