How to ignore some values from a field in splunk

[VS0909]

There is a field "Message" which contains  "Error 1 , profileid = a, jsessionid=b"

I want my search query to ignore profileid and jsessionid, and dispaly error count. 

For below messages in logs, query should display error1 count as 2 and error2 count as 1

Message 1 : Error 1 , profileid = a, jsessionid=b

Message 2 : Error 1, profileid = c, jsessionid=d

Message 3 : Error 2 , profileid = e, jsessionid=f

Please help

[VS0909]

@isoutamo  @gcusello  - Appreciate all your efforts for replying to my query. Thanks again.

[gcusello]

Hi @VS0909,

you could extract the erro count field by the message field using the rex command:

| rex field=message "^(?<error_count>[^,]*)"

that you can test at https://regex101.com/r/tKeHuI/1

Ciao.

Giuseppe

[VS0909]

@gcusello  Thanks for the reply!

Is there any way in which I can ignore  "profileid" and "jsessionid" values while doing a search query on "message", as profileid and jsessionid are also fields.

I am trying below query, but not able to ignore "profileid" and "jsessionid" field values.

index=serverlogs1 log_level = error | fields -profileid -jsessionid | stats count by message. 

Please help how to ignore  "profileid" and "jsessionid" field values.

[gcusello]

Hi @VS0909,

if you want to ignore a field, you have to put a space between "-" and the field name:

| fields - profileid - jsessionid

but in this way you only don't display them.

Instead you need to search only for the first part of the message field and the only way is to extract the relevant part of the message field using the regex I hinted:

index=serverlogs1 log_level=error 
| rex field=message "^(?<error>[^,]*)"
| stats count BY error 

Then use the extracted field for the search.

Ciao.

Giuseppe 

[VS0909]

@gcusello Thanks for the reply! Appreciate it

For my scenario, the message will not always be in the same format.

My goal is to scan splunk logs for all errors in stack trace for past one week (excluding today) and find any anomaly/ new errors in past 24 hours.

Most of the times "message" field have stack trace.

But, for same error  in "message" field  profileid /jsession fields should be ignored.

On using below query, then  for same error, if jsessionid and profile are different it shows that also  as an anomaly (which is not what I expected)

 "index=serverlogs1 log_level=error  | anomalydetection action=filter message  "

Please help!

[gcusello]

Hi @VS0909,

to scan last week and compare found errors with the last 24 hours errors isn't a problem.

As I said, the main problem is to identify errors: if you can identify them it's easy, if not it's a problem.

So I hint to concentrate your effort to this: it's possible to exclude the parte of message field containing profileid and jsession, but in anyway you have to identify the error then you can compare results using the answer of your previous question.

Ciao.

Giuseppe

[VS0909]

@gcusello  Thanks a lot for the reply! Appreciate it

I am executing below splunk query, by using rex to get first line of the "message" field. Query returns result for "message" field. But, on executing below query, there is no error in query execution and also no results are returned.

index=serverlogs1 log_level=error

| rex field= message"(?<errorDetails>\A.*)"

| stats count by errorDetails

Please help!

[gcusello]

Hi @VS0909,

your regex isn't correct:

• instead \A use ^.
• then ^ must be outside the field definition.

Why the one I hinted isn't correct for you?

Anyway, test it at https://regex101.com/r/tKeHuI/1

Ciao.

Giuseppe

[VS0909]

@gcusello  Thanks for the reply!

My search criteria is to select the whole first line as per the updated message field.

Updated message filed is something like:

Message 1:

Error 1
profileid = a, jsessionid=b

(?<errorDetails>\A.*) returns me first line as per - https://regex101.com/r/tKeHuI/1

Then, when trying to execute below query, there is no error in query execution and but also no results are returned. (though "message" field have data)

index=serverlogs1 log_level=error

| rex field= message"(?<errorDetails>\A.*)"

| stats count by errorDetails

Please help!

[gcusello]

Hi @VS0909,

Let me understand: you don't have the comma after the error message and you have profileid and jsessionid in a different row.

Try this regex:

| rex field=message "^(?<error_count>.*)\s+profileid"

that you can test at https://regex101.com/r/tKeHuI/2

In other words, you have to identify the possible cases and find the relative regex.

Ciao.

Giuseppe

[VS0909]

@gcusello  Thanks for the reply!

I have got the required rex and when executing below , there is no error in query execution and but also no results are returned. (though "message" field have data)

index=serverlogs1 log_level=error

| rex field= message"(?<errorDetails>REGEX Expression)"

| stats count by errorDetails

I checked REGEX Expression, it is correct as per my requirement, but on executing query with stats command, does not return any data, though no error while executing.

[gcusello]

Hi @VS0909,

to debug your search, run this search:

index=serverlogs1 log_level=error
| rex field=message "(?<errorDetails>REGEX Expression)"
| table message errorDetails

and see which values you have for message and errorDetails fields, and if you have both.

Sometimes (it's rare but it happens) there's a difference between Splunk and regex101.

Ciao.

Giuseppe

[VS0909]

@gcusello  Thanks for the reply!

There are below 2 cases

1) Without spaces between message and " like  - message"

.......| rex field=message"(?<errorDetails>REGEX Expression)"
| table message errorDetails

This gives me empty values for errorDetails but gives data for  message.

2) With spaces between message and " like  - message "

.......| rex field=message "(?<errorDetails>REGEX Expression)"
| table message errorDetails

This gives error while executing query in splunk "Error in 'rex' command: The regex 'message' does not extract anything. It should specify at least one named group. Format: (?<name>...)."

As per my updated message requirement  the correct rex is  (?<errorDetails>\A.*) , which works fine in regex101, which just extracts first line. I just need the first line, and nothing from below lines.

Message:

Error 1
profileid = a, jsessionid=b

Please help!

[gcusello]

Hi @VS0909,

please share both the kind of messages.

Ciao.

Giuseppe

[isoutamo]

Hi

this probably solve your issue?

https://community.splunk.com/t5/Splunk-Search/How-to-edit-my-regular-expression-to-grab-the-first-li...

r. Ismo