topic Re: How to search count of multiple logs in Splunk Search

How to search count of multiple logs?

vishal_pcap — Tue, 17 Jan 2023 19:29:23 GMT

How can I write a query like following?

index=my_app
| eval userError="Error while fetching User"
| eval addressError = "Did not find address of user"
| stats count(userError) as totalUserErrors, count(addressError) as totalAddressErrors

Expected output:

Error while fetching User	50
Did not find address of user	30

Re: How to search count of multiple logs

ITWhisperer — Mon, 16 Jan 2023 14:11:23 GMT

Is this what you are trying to do?

index=my_app | stats count(eval(userError=="Error while fetching User")) as totalUserErrors, count(eval(addressError=="Did not find address of user")) as totalAddressErrors

Re: How to search count of multiple logs

vishal_pcap — Mon, 16 Jan 2023 14:23:16 GMT

The query doesn't return anything 😞 I wanted to have a tabular output - error message and number of times it appeared

Re: How to search count of multiple logs

ITWhisperer — Mon, 16 Jan 2023 14:26:35 GMT

Please share some of your events (in a code block using the </> formatting button), anonymised of course.

Also, share your current search, so we can see what you have tried so far..

Re: How to search count of multiple logs

vishal_pcap — Mon, 16 Jan 2023 14:53:05 GMT

So when I search the following query in splunk, it returns 50k+ records

index=xxxeks_prod_app cluster_name="xxxx-xxxxx-prod-eks-cluster-v1" container_name="xx*-service" "Error while fetching Users"

and I want to see the multiple error logs and their count (for the duration I have selected e.g. 30 minutes)
Here's what I have tried but it returned 0 count

index=xxxeks_prod_app cluster_name="xxxx-xxxxx-prod-eks-cluster-v1" container_name="xx*-service" | stats count(eval(userError=="Error while fetching Users")) as totalUserErrors

Re: How to search count of multiple logs

ITWhisperer — Mon, 16 Jan 2023 15:03:31 GMT

It looks like the errorError field has not been extracted.

Re: How to search count of multiple logs

vishal_pcap — Mon, 16 Jan 2023 15:10:56 GMT

How to extract that?

Re: How to search count of multiple logs

ITWhisperer — Mon, 16 Jan 2023 15:13:11 GMT

Please share some of your events (in a code block using the </> formatting button), anonymised of course.

Re: How to search count of multiple logs

vishal_pcap — Mon, 16 Jan 2023 15:26:03 GMT

oh yes,

1/16/23 7:15:44.624 AM 2023-01-16 07:15:44 AM [http-nio-8080-exec-8] [trace_id: / span_id: ] ERROR jobTraceId= commandTraceId= {X-B3-ParentSpanId=xxxxxx, X-B3-SpanId=xxxxx, X-B3-TraceId=xxxxx, X-Span-Export=false, parentId=xxxxx, spanExportable=false, spanId=xxxx, traceId=xxxxxxxxxxxx} com.demo.controller.UserController - Error while fetching Users participant and plan info details=Could not find any User for the userId=202961636 java.lang.IllegalArgumentException: Could not find any User for userId=202961636 at com.demo.service.UserServiceV2.lambda$prepareUserInfo$4(UserServiceV2.java:520) ~[demo-data-rest-1.0.22.12.40.jar:?] at java.util.Optional.orElseThrow(Unknown Source) ~[?:?] at com.demo.service.UserServiceV2.prepareUserInfo(UserServiceV2.java:520) ~[demo-data-rest-1.0.22.12.40.jar:?] host = ip-11-000-00-00.us-west-2.compute.internalsource = /var/log/containers/demo-v2-service-55f87cc4v2-v2service-78f8e0f8ff9689627faa4718f34578bd511913596cbf57.logsourcetype = kube:container:demo-v2-service

and

2023-01-16 07:21:28 AM [http-nio-8080-exec-63] [trace_id: / span_id: ] ERROR com.demo.service.PersonApiChunkService - Error while handling user: bae9877cf5ab433xx39fda32ffd9833exx6bf2 com.demo.exception.ResourceNotFoundException: No User address Found for personId=329813370 deviceId=7440501_P_192 host = ip-11-000-00-00.us-west-2.compute.internalsource = /var/log/containers/demo-v2-service-55f87cc4v2-v2service-78f8e0f8ff9689627faa4718f34578bd511913596cbf57.logsourcetype = kube:container:demo-v2-service

Re: How to search count of multiple logs

ITWhisperer — Mon, 16 Jan 2023 15:54:07 GMT

Given that your event don't appear to have any structured fields, you could try counting matches of the _raw field

| stats count(eval(match(_raw,"Error while fetching Users"))) as userError count(eval(match(_raw,"No User address Found"))) as addressError

Re: How to search count of multiple logs

vishal_pcap — Mon, 16 Jan 2023 16:09:40 GMT

This works! 🙂 Thank you very much, I have also figured out one more way to do this:

| eval errorType=case( match(_raw, "Error while fetching Users"), "Error while fetching Users", match(_raw, "No User address Found"), "No User address Found" ) | stats count by errorType | table errorType, count

This gives me the table structure I wanted, the error message and count.