Is it possible for this search to include present errors and also dynamically add new error added by developer?

[ps2019]

Hi guys,

I need help with a Splunk query. The boss wants me to have a total of all different types of errors. 

When I run this query:

index = css-dev error = "*"

it gives the logs where for each log there is an error field present. The error field has 5 values - access_denied, invalid_request, invalid_token, server_error, unauthorised_client.

In addition to this "error" field, there are some other errors also which I want to capture but they are added by developers by adding them using log.

These errors are:

1. runtime error: attempt to get length of a boolean value

2. Authentication error : WRONGPASS invalid username-password pair

3. Error while sending 2 (size = 1KB) traces to the DD agent

So these above 3 errors are not included in the "error" field and so therefore when i run the query - index = css-dev error="*" , I cannot find the above 3 errors.

What I want is a query that should include the already present errors in the "error" field(access_denied, invalid_request, invalid_token, server_error, unauthorised_client), and should  also dynamically add any new error added by the developer.

Is it possible?

[PickleRick]

Well, you can find something but you have to know what it is. If your developers keep adding various errors in various parts of the event and in various formats - well, splunk can't guess which part is the name of the error.

How do you expect a solution (not even Splunk, any solution) to guess which part identifies an error from those events?

(WW) warning, (EE) error, (NI) not implemented, (??) unknown.
Oct 13 15:16:04.091362: whack error: Broken pipe (errno 32)
2022-09-24T15:31:40+0200 ERROR Error in POSTIN scriptlet in rpm package kmod-v4l2loopback-6.0.0-0.rc6.20220922gitdc164f4fb00a.43.vanilla.1.fc37.x86_64
2022-09-30T06:25:07+0200 INFO Error during transfer: Curl error (6): Couldn't resolve host name for https://mirrors.fedoraproject.org/metalink?repo=updates-testing-f37&arch=x86_64 [Could not resolve host: mirrors.fedoraproject.org]
05:45.812398 lvcreate[3460] error/errseg.c:105 Initialised segtype: error

And these are just excerpts from my laptop's logs.

If your logs have no common structure whatsoever and furthermore you want a general solution that includes properly extracting a field (because that's what it boils down to) in a completely undefined way... well, sorry. That can't be done.

[ps2019]

I only want that the 2 errors:

1. Authentication error :

2. runtime error:

also get added to the list of errors, the way of identifying them would be after error there would be a ":"(colon) present directly after the word like this - error:

or after a space like - error :

So can't we modify the regex or the query in a way so as to achieve this?

[PickleRick]

This can be easily matched by something like

[Ee]rror\s*:\s*(?<error_message>.*)

But this will capture everything up to the end of the line so if you have anything allowing you to liimt the error message, not just capture any string, it would be helpful.

[johnhuang]

The key is making sure your search results includes the events in question. Then write a regex to extract the custom error values.

The regex extraction in this solution is not perfect since it's based on the incomplete data you've provided.

index="css-dev" (error=* OR error)
| rex "(?<custom_error>(\w*\s)?(E|e)rror\s?\:?(\s|\w|\-|\(|\=|\))*)"
| eval error_name=COALESCE(error, custom_error)
| where LEN(error_name)>1
| stats count AS event_count by error_name

[ps2019]

When i run this query:

index="css-dev" (source != "*qa*" AND source = "*pinxtgateway*") (error=* OR error)
| rex "(?<custom_error>(\w*\s)?(E|e)rror\s?\:?(\s|\w|\-|\(|\=|\))*)"
| eval error_name=COALESCE(error, custom_error)
| where LEN(error_name)>1
| stats count AS event_count by error_name

it gives me this:

error_name                                                                              event_count

Error                                                                                                 21478                      

Error:                                                                                                22                   

Server Error</                                                                              21                

access_denied                                                                             1014        

error                                                                                                   4         

error":                                                                                                 85                      

error]                                                                                                 1425919                              

error ] [                                                                                               179                                              

error],                                                                                                   85                                                  

error]}"                                                                                                170                                            

invalid_request                                                                                  22                                    

invalid_token                                                                                        4                                                

open error                                                                                             179                                    

server_error                                                                                           86               

But I do not want like this. 

[johnhuang]

Responding to your message. You can try one of the 2 regex below and see which ones gives you the best result. The 2nd regex extracts everything after the error, so this improves flexiblity but could give you more noise.

| rex "[^\[](?<custom_error>(\w*\s)?(E|e)rror\s?\:?(\s|\w|\-|\(|\=|\)|\.)*)"

| rex "[^\[](?<custom_error>(\w*\s)?(E|e)rror.*)"

After you extract the errors, you can clean up the results further by removing unwanted details, for example:

Input:

Error while sending 1 (size=1024B) traces to the DD agent
Error while sending 1 (size=1KB) traces to the DD agent
Error while sending 1 (size=550B) traces to the DD agent

| rex field=custom_error mode=sed "s/\d\s\(size\=\w*\)\s//"

Output:

Error while sending traces to the DD agent

[johnhuang]

The regex error extraction needs to be improved. I'd be happy to help you with that, but you have to provide some examples of the actual data (please review to make sure there's no sensitive information).

You can post the results of the query here to send it to me in a message.

index="css-dev" (source != "*qa*" AND source = "*pinxtgateway*") (error=* OR error)
| rex "(?<custom_error>(\w*\s)?(E|e)rror\s?\:?(\s|\w|\-|\(|\=|\))*)"
| eval error_name=COALESCE(error, custom_error)
| search NOT error_name="*_*"
| dedup error_name
| table error_name _raw

[ps2019]

What I want is something like this:

error_name                          event_count

access_denied                      (count)

invalid_request                      (count)

invalid_token                           (count)

server_error                             (count)

unauthorised_client              (count)

Authorisation error :               (count)

runtime error:                            (count)

xyz error                                       (count)

In future whenever any new error comes, it should then add that new error to this format with its count

error field has 5 values already - access_denied, invalid_request, invalid_token, server_error, unauthorised_client

This error field does not include the errors set by developer like - Authorisation error, runtime error, etc.

These errors set by developer should also be added to the list and all the errors should be under one column