topic Re: Why do special characters "[0[0m" appear in my events? in Getting Data In

Why do special characters "[0[0m" appear in my events?

pmerlin1 — Thu, 01 Dec 2016 14:49:41 GMT

I deploy Splunk forwarder on a JBoss server to forward data towards my test environment Splunk.

In the Universal Forwarder (UF) monitor file server.log file, the line

01/12/16 15:11:50,398 INFO  [org.jboss.as] (MSC service thread 1-3) JBAS015950: JBoss EAP 6.4.8.GA (AS 7.5.8.Final-redhat-2) stopped in 358ms

is transfomed by the event below

[0m[0m01/12/16 15:11:50,398 INFO  [org.jboss.as] (MSC service thread 1-3) JBAS015950: JBoss EAP 6.4.8.GA (AS 7.5.8.Final-redhat-2) stopped in 358ms

All lines is prepended by characters [0m[0m for INFO message or [0[31m when it's a ERROR message

Someone can explain why?

Re: Why do special characters "[0[0m" appear in my events?

ddrillic — Thu, 01 Dec 2016 16:25:26 GMT

Interesting, we see these characters here as well - archive.travis-ci.org

Re: Why do special characters "[0[0m" appear in my events?

gokadroid — Thu, 01 Dec 2016 19:18:29 GMT

A shot in the dark will be the log lines might have colour coding info in them to appear colourful on console, however when they are sent over to Splunk this colour coding stays preserved as part of log line. Generally the colour code numbers are as follows clubbed together with [ and m:

Black        0;30     Dark Gray     1;30
Red          0;31     Light Red     1;31
Green        0;32     Light Green   1;32
Brown/Orange 0;33     Yellow        1;33
Blue         0;34     Light Blue    1;34
Purple       0;35     Light Purple  1;35
Cyan         0;36     Light Cyan    1;36
Light Gray   0;37     White         1;37

You can try to rex them out using something like this:

your query to return the event
| rex mode=sed field=_raw "s/\[[0-9]+m//g"
| complete your query

Re: Why do special characters "[0[0m" appear in my events?

pmerlin1 — Thu, 01 Dec 2016 22:42:23 GMT

Thank you gokrakoid
I prefer to strip the color codes before indexing
I try in transforms.conf on my indexer :
[MyLOG-stripper-colorcodes]
REGEX = ^[\d+m[\d+m(.*)$
FORMAT = $1
DEST_KEY = _raw

but it deosn't work. The characters are not removed still. Any idea ?

Re: Why do special characters "[0[0m" appear in my events?

gokadroid — Thu, 01 Dec 2016 23:00:10 GMT

Can you try to escape any special characters like [ to make it \[ or try to see if sedcmd can assist to do that in props.conf?

[your:sourcetype]
  SEDCMD-rep_1 = s/\[[0-9]+m//g

Please note that you will have other properties as well in props.conf so do not remove them for your:sourcetype. Just append the line SEDCMD-rep_1 = s/\[[0-9]+m//g to existing properties.

Re: Why do special characters "[0[0m" appear in my events?

pmerlin1 — Fri, 02 Dec 2016 09:18:48 GMT

Thanks gokadroit.
I solve my color syntaxing issue, in modfying my custom java log handler . I dont want report the clean effort on Splunk. I remove %K{level}% of the pattern in the logger configuration.
Now it works the event is display without color codes.

Re: Why do special characters "[0[0m" appear in my events?

mahendra5 — Fri, 14 Jul 2017 21:58:35 GMT

This solution is working for us very well.

I can combine it with my query and rex does the job. I can create macros too.

| rex mode=sed field=_raw "s/[[0-9]+m//g"

Re: Why do special characters "[0[0m" appear in my events?

mahendra5 — Fri, 14 Jul 2017 22:17:59 GMT

@gokadroid
this works perfect, sourectype can easily eliminate those before even it gets indexed.