How to extract fields from log message?

[Anu1184]

Hi , I am trying to create a dashboard having stats summary basis on - error log resides into log message. 
Can someone help , how can I extract respective text and mark them as a field to get count for final stats -

Sample Event Data :

2023-05-12 09:48:30,580 - abc_sdk._internal.worker - INFO - _worker_process - request_id=xyz-4fa1-b28a-0b62e4199a53 - x-gw-ims-client-id=X_webapp x-gw-ims-user-id=abc@xyz.com
Total time taken: 63.58056879043579 can not stack video

#Total requests , #Total errors for unstacked videos.

[ITWhisperer]

How do you distinguish between errors and non-errors?

[Anu1184]

If a message has failure string like - "failed in processing" then it will be an error , if there is no as such string message present into log , then the message seems went fine.

[yuanliu]

This is still too vague, meaning that there are too many possible paths that do not meet your specific requirements.  Remember, you know your dataset, and you know the specific problem you want to solve.  Unless you can explain these to volunteers here, we can only speculate - most of the time incorrectly.

For example, "#Total requests , #Total errors for unstacked videos" can be interpreted in any of alternative ways

1. a count of all requests in which some videos are stacked, some are unstacked, but no matter whether there is an error, then a count of requests in which video is not stacked AND in which there is an error.
2. a count of only those requests in which videos are unstacked, but no matter whether there is an error, then a count of requests in which video is unstacked AND in which an error exists.

In addition, you give only one event sample, in which "failed in processing" is not present.  What is the criteria of "unstacked videos"?  The sample event only contains this string "can not stack video."  Without deep domain knowledge that you possess, the volunteer can easily make a mistake in presuming that "can not stack video" means "unstacked video".

Even if it is correct to assume that "can not stack video" is semantically identical to "unstacked video", what about the loglevel INFO in the sample event that also does NOT contain the error string "failed in processing."  It is quite logical to assume that when there is an error, the loglevel should be ERROR rather than INFO.  If this is the case, the solution would be much simpler than otherwise.

It is even logical to assume that, when the log source was set up, basic fields such as loglevel is already extracted. (Splunk comes with a standard transformation that you can adopt and apply to a source/sourcetype.)

Here, I will illustrate one of the many possible combinations with the following assumptions:

• Your requirement is a count of only those requests in which videos are unstacked, but no matter whether there is an error, then a count of requests in which video is unstacked AND in which an error exists.
• "can not stack video" is semantically identical to "unstacked video".
• ERROR is a valid loglevel in addition to INFO.
• "errors for unstacked videos" can be discerned by loglevel ERROR.
• loglevel is sandwiched between two spaces, the combination of which is sandwiched between two dashes (-).
• loglevel is not automatically extracted.

"can not stack video" ``` only retrieve events with unstacked videos ```
| rex "- (?<loglevel>(FATAL|ERROR|WARN|INFO|DEBUG|TRACE)) -"
``` "failed in processing" is not ued in favor of loglevel ERROR ```
| stats count as "#Total requests" count(eval(loglevel == "ERROR")) as "#Total errors for unstacked videos"