Hi , I am trying to create a dashboard having stats summary basis on - error log resides into log message.
Can someone help , how can I extract respective text and mark them as a field to get count for final stats -
Sample Event Data :
2023-05-12 09:48:30,580 - abc_sdk._internal.worker - INFO - _worker_process - request_id=xyz-4fa1-b28a-0b62e4199a53 - x-gw-ims-client-id=X_webapp x-gw-ims-user-id=abc@xyz.com
Total time taken: 63.58056879043579 can not stack video
#Total requests , #Total errors for unstacked videos.
How do you distinguish between errors and non-errors?
If a message has failure string like - "failed in processing" then it will be an error , if there is no as such string message present into log , then the message seems went fine.
This is still too vague, meaning that there are too many possible paths that do not meet your specific requirements. Remember, you know your dataset, and you know the specific problem you want to solve. Unless you can explain these to volunteers here, we can only speculate - most of the time incorrectly.
For example, "#Total requests , #Total errors for unstacked videos" can be interpreted in any of alternative ways
In addition, you give only one event sample, in which "failed in processing" is not present. What is the criteria of "unstacked videos"? The sample event only contains this string "can not stack video." Without deep domain knowledge that you possess, the volunteer can easily make a mistake in presuming that "can not stack video" means "unstacked video".
Even if it is correct to assume that "can not stack video" is semantically identical to "unstacked video", what about the loglevel INFO in the sample event that also does NOT contain the error string "failed in processing." It is quite logical to assume that when there is an error, the loglevel should be ERROR rather than INFO. If this is the case, the solution would be much simpler than otherwise.
It is even logical to assume that, when the log source was set up, basic fields such as loglevel is already extracted. (Splunk comes with a standard transformation that you can adopt and apply to a source/sourcetype.)
Here, I will illustrate one of the many possible combinations with the following assumptions:
"can not stack video" ``` only retrieve events with unstacked videos ```
| rex "- (?<loglevel>(FATAL|ERROR|WARN|INFO|DEBUG|TRACE)) -"
``` "failed in processing" is not ued in favor of loglevel ERROR ```
| stats count as "#Total requests" count(eval(loglevel == "ERROR")) as "#Total errors for unstacked videos"