Dashboards & Visualizations

How to not show particular Error message in a splunk alert

aditsss
Builder

Hi Everyone,

I am creating one alert:

The search query is below:

index=abc  ns=blazegateway ERROR|rex field=_raw "ERROR(?<Error_Message>.*)" |eval _time = strftime(_time,"%Y-%m-%d %H:%M:%S.%3N")| eval Error_Message=if(Error_Message="\",",null,Error_Message)|cluster showcount=t t=0.2|table app_name, Error_Message ,cluster_count,_time, pod_name,ns |dedup Error_Message| rename app_name as APP_NAME, _time as Time, environment as Environment, pod_name as Pod_Name, cluster_count as Count

I need to show all the Errors so I am fetching on keyword Error.

My splunk log is below:

Xms3096m -Xmx3096m -Dsidh_psf_spring_profile=e1 -Dspring.profiles.active=e1 -Dsidh_symmetric_cipher_key=MasVU4msfPLjItTYo1VLRgfi5VjJ46axIZ/9qTUAUmY= -Dio.javaagent.slf4j.simpleLogger.defaultLogLevel=ERROR', '-jar', '/opt/app-root/app.jar']

 

Dio.javaagent.slf4j.simpleLogger.defaultLogLevel=ERROR

But the ERROR in above logs are not ERROR.

I dont want them to come in alerts.

what changes I should do in my search query to not get them but get rest of the Errors.

 

Thanks in advance

Labels (3)
0 Karma

rnowitzki
Builder

Hi @aditsss,

You could change your rex command to this:

|rex field=_raw "\sERROR(?<Error_Message>.*)"


Note that I added \s before ERROR, because usually in an Error Message/Event the string "ERROR" is not directly located next to another string . You could also add a \s after the "ERROR".

Hope that helps.
BR
Ralph

--
Karma and/or Solution tagging appreciated.
0 Karma

ITWhisperer
Ultra Champion

It looks like you are checking is Error_Message is a double quote and a comma (Error_Message="\",") but the log appears to show single quote (defaultLogLevel=ERROR', ). 

 

0 Karma

aditsss
Builder

@ITWhisperer @rnowitzki 

How should I change my rex command in this case

0 Karma

ITWhisperer
Ultra Champion

You could include a negative lookback to ensure ERROR isn't preceded by LogLevel=

|rex field=_raw "(?<!LogLevel=)ERROR(?<Error_Message>.*)" 

rnowitzki
Builder

The eval statement that @ITWhisperer is refering to would set the Error_Message to null if it is <double quote><comma>,  you could change that to 

if(Error_Message="\',",null,Error_Message).


But if you use my adjusted rex command, you would not even have this "wrong" Error Messages in your search results.

You just need to check that your valid Error Events have the string "ERROR" with space in front, which is usually the case.

--
Karma and/or Solution tagging appreciated.