How to not show particular Error message in a splunk alert

[aditsss]

Hi Everyone,

I am creating one alert:

The search query is below:

index=abc  ns=blazegateway ERROR|rex field=_raw "ERROR(?<Error_Message>.*)" |eval _time = strftime(_time,"%Y-%m-%d %H:%M:%S.%3N")| eval Error_Message=if(Error_Message="\",",null,Error_Message)|cluster showcount=t t=0.2|table app_name, Error_Message ,cluster_count,_time, pod_name,ns |dedup Error_Message| rename app_name as APP_NAME, _time as Time, environment as Environment, pod_name as Pod_Name, cluster_count as Count

I need to show all the Errors so I am fetching on keyword Error.

My splunk log is below:

Xms3096m -Xmx3096m -Dsidh_psf_spring_profile=e1 -Dspring.profiles.active=e1 -Dsidh_symmetric_cipher_key=MasVU4msfPLjItTYo1VLRgfi5VjJ46axIZ/9qTUAUmY= -Dio.javaagent.slf4j.simpleLogger.defaultLogLevel=ERROR', '-jar', '/opt/app-root/app.jar']

Dio.javaagent.slf4j.simpleLogger.defaultLogLevel=ERROR

But the ERROR in above logs are not ERROR.

I dont want them to come in alerts.

what changes I should do in my search query to not get them but get rest of the Errors.

Thanks in advance

[rnowitzki]

Hi @aditsss,

You could change your rex command to this:

|rex field=_raw "\sERROR(?<Error_Message>.*)"

Note that I added \s before ERROR, because usually in an Error Message/Event the string "ERROR" is not directly located next to another string . You could also add a \s after the "ERROR".

Hope that helps.
BR
Ralph

[ITWhisperer]

It looks like you are checking is Error_Message is a double quote and a comma (Error_Message="\",") but the log appears to show single quote (defaultLogLevel=ERROR', ). 

[aditsss]

@ITWhisperer @rnowitzki 

How should I change my rex command in this case

[ITWhisperer]

You could include a negative lookback to ensure ERROR isn't preceded by LogLevel=

|rex field=_raw "(?<!LogLevel=)ERROR(?<Error_Message>.*)" 

[rnowitzki]

The eval statement that @ITWhisperer is refering to would set the Error_Message to null if it is <double quote><comma>,  you could change that to 

if(Error_Message="\',",null,Error_Message).

But if you use my adjusted rex command, you would not even have this "wrong" Error Messages in your search results.

You just need to check that your valid Error Events have the string "ERROR" with space in front, which is usually the case.