topic Re: How to ignore some values from a field in splunk in Splunk Search

How to ignore some values from a field in splunk

VS0909 — Tue, 01 Sep 2020 06:15:42 GMT

There is a field "Message" which contains "Error 1 , profileid = a, jsessionid=b"

I want my search query to ignore profileid and jsessionid, and dispaly error count.

For below messages in logs, query should display error1 count as 2 and error2 count as 1

Message 1 : Error 1 , profileid = a, jsessionid=b

Message 2 : Error 1, profileid = c, jsessionid=d

Message 3 : Error 2 , profileid = e, jsessionid=f

Please help

Re: How to ignore some values from a field in splunk

gcusello — Tue, 01 Sep 2020 07:08:39 GMT

Hi @VS0909,

you could extract the erro count field by the message field using the rex command:

| rex field=message "^(?<error_count>[^,]*)"

that you can test at https://regex101.com/r/tKeHuI/1

Ciao.

Giuseppe

Re: How to ignore some values from a field in splunk

VS0909 — Tue, 01 Sep 2020 07:18:24 GMT

@gcusello Thanks for the reply!

Is there any way in which I can ignore "profileid" and "jsessionid" values while doing a search query on "message", as profileid and jsessionid are also fields.

I am trying below query, but not able to ignore "profileid" and "jsessionid" field values.

index=serverlogs1 log_level = error | fields -profileid -jsessionid | stats count by message.

Please help how to ignore "profileid" and "jsessionid" field values.

Re: How to ignore some values from a field in splunk

gcusello — Tue, 01 Sep 2020 07:24:24 GMT

Hi @VS0909,

if you want to ignore a field, you have to put a space between "-" and the field name:

| fields - profileid - jsessionid

but in this way you only don't display them.

Instead you need to search only for the first part of the message field and the only way is to extract the relevant part of the message field using the regex I hinted:

index=serverlogs1 log_level=error | rex field=message "^(?<error>[^,]*)" | stats count BY error

Then use the extracted field for the search.

Ciao.

Giuseppe

Re: How to ignore some values from a field in splunk

VS0909 — Tue, 01 Sep 2020 08:20:10 GMT

@gcusello Thanks for the reply! Appreciate it

For my scenario, the message will not always be in the same format.

My goal is to scan splunk logs for all errors in stack trace for past one week (excluding today) and find any anomaly/ new errors in past 24 hours.

Most of the times "message" field have stack trace.

But, for same error in "message" field profileid /jsession fields should be ignored.

On using below query, then for same error, if jsessionid and profile are different it shows that also as an anomaly (which is not what I expected)

"index=serverlogs1 log_level=error | anomalydetection action=filter message "

Please help!

Re: How to ignore some values from a field in splunk

gcusello — Tue, 01 Sep 2020 08:34:41 GMT

Hi @VS0909,

to scan last week and compare found errors with the last 24 hours errors isn't a problem.

As I said, the main problem is to identify errors: if you can identify them it's easy, if not it's a problem.

So I hint to concentrate your effort to this: it's possible to exclude the parte of message field containing profileid and jsession, but in anyway you have to identify the error then you can compare results using the answer of your previous question.

Ciao.

Giuseppe

Re: How to ignore some values from a field in splunk

VS0909 — Tue, 01 Sep 2020 14:26:33 GMT

@gcusello Thanks a lot for the reply! Appreciate it

I am executing below splunk query, by using rex to get first line of the "message" field. Query returns result for "message" field. But, on executing below query, there is no error in query execution and also no results are returned.

index=serverlogs1 log_level=error

| rex field= message"(?<errorDetails>\A.*)"

| stats count by errorDetails

Please help!

Re: How to ignore some values from a field in splunk

gcusello — Tue, 01 Sep 2020 14:41:17 GMT

Hi @VS0909,

your regex isn't correct:

instead \A use ^.
then ^ must be outside the field definition.

Why the one I hinted isn't correct for you?

Anyway, test it at https://regex101.com/r/tKeHuI/1

Ciao.

Giuseppe

Re: How to ignore some values from a field in splunk

VS0909 — Tue, 01 Sep 2020 14:55:25 GMT

@gcusello Thanks for the reply!

My search criteria is to select the whole first line as per the updated message field.

Updated message filed is something like:

Message 1:

Error 1
profileid = a, jsessionid=b

(?<errorDetails>\A.*) returns me first line as per - https://regex101.com/r/tKeHuI/1

Then, when trying to execute below query, there is no error in query execution and but also no results are returned. (though "message" field have data)

index=serverlogs1 log_level=error

| rex field= message"(?<errorDetails>\A.*)"

| stats count by errorDetails

Please help!

Re: How to ignore some values from a field in splunk

gcusello — Tue, 01 Sep 2020 15:13:25 GMT

Hi @VS0909,

Let me understand: you don't have the comma after the error message and you have profileid and jsessionid in a different row.

Try this regex:

| rex field=message "^(?<error_count>.*)\s+profileid"

that you can test at https://regex101.com/r/tKeHuI/2

In other words, you have to identify the possible cases and find the relative regex.

Ciao.

Giuseppe

Re: How to ignore some values from a field in splunk

VS0909 — Tue, 01 Sep 2020 15:23:50 GMT

@gcusello Thanks for the reply!

I have got the required rex and when executing below , there is no error in query execution and but also no results are returned. (though "message" field have data)

index=serverlogs1 log_level=error

| rex field= message"(?<errorDetails>REGEX Expression)"

| stats count by errorDetails

I checked REGEX Expression, it is correct as per my requirement, but on executing query with stats command, does not return any data, though no error while executing.

Re: How to ignore some values from a field in splunk

gcusello — Tue, 01 Sep 2020 15:28:56 GMT

Hi @VS0909,

to debug your search, run this search:

index=serverlogs1 log_level=error | rex field=message "(?<errorDetails>REGEX Expression)" | table message errorDetails

and see which values you have for message and errorDetails fields, and if you have both.

Sometimes (it's rare but it happens) there's a difference between Splunk and regex101.

Ciao.

Giuseppe

Re: How to ignore some values from a field in splunk

VS0909 — Tue, 01 Sep 2020 15:44:42 GMT

@gcusello Thanks for the reply!

There are below 2 cases

1) Without spaces between message and " like - message"

.......| rex field=message"(?<errorDetails>REGEX Expression)"
| table message errorDetails

This gives me empty values for errorDetails but gives data for message.

2) With spaces between message and " like - message "

.......| rex field=message "(?<errorDetails>REGEX Expression)"
| table message errorDetails

This gives error while executing query in splunk "Error in 'rex' command: The regex 'message' does not extract anything. It should specify at least one named group. Format: (?<name>...)."

As per my updated message requirement the correct rex is (?<errorDetails>\A.*) , which works fine in regex101, which just extracts first line. I just need the first line, and nothing from below lines.

Message:

Error 1
profileid = a, jsessionid=b

Please help!

Re: How to ignore some values from a field in splunk

isoutamo — Tue, 01 Sep 2020 15:55:26 GMT

this probably solve your issue?

https://community.splunk.com/t5/Splunk-Search/How-to-edit-my-regular-expression-to-grab-the-first-line-in-a/td-p/266766

r. Ismo

Re: How to ignore some values from a field in splunk

gcusello — Wed, 02 Sep 2020 06:23:04 GMT

Hi @VS0909,

please share both the kind of messages.

Ciao.

Giuseppe

Re: How to ignore some values from a field in splunk

VS0909 — Fri, 04 Sep 2020 13:50:32 GMT

@isoutamo @gcusello - Appreciate all your efforts for replying to my query. Thanks again.