topic Is it possible for this search to include present errors and also dynamically add new error added by developer? in Splunk Search

Is it possible for this search to include present errors and also dynamically add new error added by developer?

ps2019 — Wed, 12 Oct 2022 19:41:51 GMT

Hi guys,

I need help with a Splunk query. The boss wants me to have a total of all different types of errors.

When I run this query:

index = css-dev error = "*"

it gives the logs where for each log there is an error field present. The error field has 5 values - access_denied, invalid_request, invalid_token, server_error, unauthorised_client.

In addition to this "error" field, there are some other errors also which I want to capture but they are added by developers by adding them using log.

These errors are:

1. runtime error: attempt to get length of a boolean value

2. Authentication error : WRONGPASS invalid username-password pair

3. Error while sending 2 (size = 1KB) traces to the DD agent

So these above 3 errors are not included in the "error" field and so therefore when i run the query - index = css-dev error="*" , I cannot find the above 3 errors.

What I want is a query that should include the already present errors in the "error" field(access_denied, invalid_request, invalid_token, server_error, unauthorised_client), and should also dynamically add any new error added by the developer.

Is it possible?

Re: Is it possible for this search to include present errors and also dynamically add new error added by developer?

johnhuang — Wed, 12 Oct 2022 20:01:03 GMT

The key is making sure your search results includes the events in question. Then write a regex to extract the custom error values.

The regex extraction in this solution is not perfect since it's based on the incomplete data you've provided.

index="css-dev" (error=* OR error) | rex "(?<custom_error>(\w*\s)?(E|e)rror\s?\:?(\s|\w|\-|\(|\=|\))*)" | eval error_name=COALESCE(error, custom_error) | where LEN(error_name)>1 | stats count AS event_count by error_name

Re: Is it possible for this search to include present errors and also dynamically add new error added by developer?

ps2019 — Thu, 13 Oct 2022 08:54:47 GMT

What I want is something like this:

error_name event_count

access_denied (count)

invalid_request (count)

invalid_token (count)

server_error (count)

unauthorised_client (count)

Authorisation error : (count)

runtime error: (count)

xyz error (count)

In future whenever any new error comes, it should then add that new error to this format with its count

error field has 5 values already - access_denied, invalid_request, invalid_token, server_error, unauthorised_client

This error field does not include the errors set by developer like - Authorisation error, runtime error, etc.

These errors set by developer should also be added to the list and all the errors should be under one column

Re: Is it possible for this search to include present errors and also dynamically add new error added by developer?

ps2019 — Thu, 13 Oct 2022 08:56:34 GMT

When i run this query:

index="css-dev" (source != "*qa*" AND source = "*pinxtgateway*") (error=* OR error)
| rex "(?<custom_error>(\w*\s)?(E|e)rror\s?\:?(\s|\w|\-|\(|\=|\))*)"
| eval error_name=COALESCE(error, custom_error)
| where LEN(error_name)>1
| stats count AS event_count by error_name

it gives me this:

error_name event_count

Error 21478

Error: 22

Server Error</ 21

access_denied 1014

error 4

error": 85

error] 1425919

error ] [ 179

error], 85

error]}" 170

invalid_request 22

invalid_token 4

open error 179

server_error 86

But I do not want like this.

Re: Is it possible for this search to include present errors and also dynamically add new error added by developer?

johnhuang — Thu, 13 Oct 2022 14:19:41 GMT

The regex error extraction needs to be improved. I'd be happy to help you with that, but you have to provide some examples of the actual data (please review to make sure there's no sensitive information).

You can post the results of the query here to send it to me in a message.

index="css-dev" (source != "*qa*" AND source = "*pinxtgateway*") (error=* OR error) | rex "(?<custom_error>(\w*\s)?(E|e)rror\s?\:?(\s|\w|\-|\(|\=|\))*)" | eval error_name=COALESCE(error, custom_error) | search NOT error_name="*_*" | dedup error_name | table error_name _raw

Re: Is it possible for this search to include present errors and also dynamically add new error added by developer?

johnhuang — Thu, 13 Oct 2022 18:23:04 GMT

Responding to your message. You can try one of the 2 regex below and see which ones gives you the best result. The 2nd regex extracts everything after the error, so this improves flexiblity but could give you more noise.

| rex "[^\[](?<custom_error>(\w*\s)?(E|e)rror\s?\:?(\s|\w|\-|\(|\=|\)|\.)*)"

| rex "[^\[](?<custom_error>(\w*\s)?(E|e)rror.*)"

After you extract the errors, you can clean up the results further by removing unwanted details, for example:

Input:

Error while sending 1 (size=1024B) traces to the DD agent
Error while sending 1 (size=1KB) traces to the DD agent
Error while sending 1 (size=550B) traces to the DD agent

| rex field=custom_error mode=sed "s/\d\s\(size\=\w*\)\s//"

Output:

Error while sending traces to the DD agent

Re: Is it possible for this search to include present errors and also dynamically add new error added by developer?

PickleRick — Thu, 13 Oct 2022 19:49:07 GMT

Well, you can find something but you have to know what it is. If your developers keep adding various errors in various parts of the event and in various formats - well, splunk can't guess which part is the name of the error.

How do you expect a solution (not even Splunk, any solution) to guess which part identifies an error from those events?

(WW) warning, (EE) error, (NI) not implemented, (??) unknown.
Oct 13 15:16:04.091362: whack error: Broken pipe (errno 32)
2022-09-24T15:31:40+0200 ERROR Error in POSTIN scriptlet in rpm package kmod-v4l2loopback-6.0.0-0.rc6.20220922gitdc164f4fb00a.43.vanilla.1.fc37.x86_64
2022-09-30T06:25:07+0200 INFO Error during transfer: Curl error (6): Couldn't resolve host name for https://mirrors.fedoraproject.org/metalink?repo=updates-testing-f37&arch=x86_64 [Could not resolve host: mirrors.fedoraproject.org]
05:45.812398 lvcreate[3460] error/errseg.c:105 Initialised segtype: error

And these are just excerpts from my laptop's logs.

If your logs have no common structure whatsoever and furthermore you want a general solution that includes properly extracting a field (because that's what it boils down to) in a completely undefined way... well, sorry. That can't be done.

Re: Is it possible for this search to include present errors and also dynamically add new error added by developer?

ps2019 — Fri, 14 Oct 2022 09:35:46 GMT

I only want that the 2 errors:

1. Authentication error :

2. runtime error:

also get added to the list of errors, the way of identifying them would be after error there would be a ":"(colon) present directly after the word like this - error:

or after a space like - error :

So can't we modify the regex or the query in a way so as to achieve this?

Re: Is it possible for this search to include present errors and also dynamically add new error added by developer?

PickleRick — Fri, 14 Oct 2022 11:10:05 GMT

This can be easily matched by something like

[Ee]rror\s*:\s*(?<error_message>.*)

But this will capture everything up to the end of the line so if you have anything allowing you to liimt the error message, not just capture any string, it would be helpful.