Getting Data In

Strip control/color codes

oreoshake
Communicator

I have log files with color codes and control characters that we'd like to strip because they clutter the search results. In a few cases, we've just removed them from the file with the commands found at http://www.commandlinefu.com/commands/view/3584/remove-color-codes-special-characters-with-sed

It used to be a one off case, but now it is the standard case. It seems there is some global setting I might be missing. Or is this something that I would have to use a transform for?

EDIT

So based on http://www.splunk.com/base/Documentation/4.1.4/Admin/Configurecharactersetencoding
which says "Splunk escapes the invalid characters as hex values (for example: "\xF3")."

It appears my file encoding was set wrong. Is there a way to just tell splunk to ignore the characters?

Tags (1)
1 Solution

Ayn
Legend

You can use the SEDCMD directive in props.conf for this.

For instance, let's call the sourcetype you want this to apply to "colorlogs". In props.conf, specify this:

[colorlogs]
SEDCMD-removecolorcodes = s/\x1B\[([0-9]{1,2}(;[0-9]{1,2})?)?[m|K]//g

After restarting Splunk, all color codes should be removed from any events of sourcetype "colorlogs" added to the index in the future.

(note: the regex itself is just the same as the one in the link you supplied, I'm taking their word for that it's actually effective in removing color codes 😉 )

View solution in original post

Ayn
Legend

You can use the SEDCMD directive in props.conf for this.

For instance, let's call the sourcetype you want this to apply to "colorlogs". In props.conf, specify this:

[colorlogs]
SEDCMD-removecolorcodes = s/\x1B\[([0-9]{1,2}(;[0-9]{1,2})?)?[m|K]//g

After restarting Splunk, all color codes should be removed from any events of sourcetype "colorlogs" added to the index in the future.

(note: the regex itself is just the same as the one in the link you supplied, I'm taking their word for that it's actually effective in removing color codes 😉 )

vchepkov
Explorer

I know it's an old post, but it helped me , but it leaves `[0;m` behind, which is 'Reset' I believe

 

0 Karma

vchepkov
Explorer

I just added additional 

SEDCMD-removereset=s/\x1B\[0;m//g

 

0 Karma

sogeniusio
Path Finder

Huge thumbs up, been at this all day!! Thanks!

0 Karma

mattymo
Splunk Employee
Splunk Employee

greetings from 2020. still working to this day. Now relevant with kubernetes logging.

thanks for all the wisdom over the years @Ayn

- MattyMo
0 Karma

Ayn
Legend

Great! 🙂 Could you please mark the answer as accepted? That way others see that the question has an accepted solution.

0 Karma

xplodersuv
New Member

Actually nevermind, it works!

0 Karma

xplodersuv
New Member

Thanks Ayn, makes sense. I put this in apps/search/local/props.conf but the existing data isn't clean. Is that an index or search time deal?

Am I missing a transform?

0 Karma

cssmdi
Explorer

Hi
The SEDCMD is done at index time bevore the events are stored. The stored events have the color codes stripped off.

Maybe there is also a problem with the order of commands. When I changed the sourcetype and deleted the color codes, I had to put the SEDCMD in props.conf before the REPORT... to change the sourcetype.

It would be helpful to have the possibility to remove color codes included directly in splunk.

0 Karma
Get Updates on the Splunk Community!

Dashboards: Hiding charts while search is being executed and other uses for tokens

There are a couple of features of SimpleXML / Classic dashboards that can be used to enhance the user ...

Splunk Observability Cloud's AI Assistant in Action Series: Explaining Metrics and ...

This is the fourth post in the Splunk Observability Cloud’s AI Assistant in Action series that digs into how ...

Brains, Bytes, and Boston: Learn from the Best at .conf25

When you think of Boston, you might picture colonial charm, world-class universities, or even the crack of a ...