topic Re: stats count by generating inconsistent result in Splunk Search

stats count by generating inconsistent result

noman377 — Fri, 07 Aug 2020 18:48:37 GMT

I have a very simple search:

index=logs_glbl sourcetype=kube:container:app-name namespace=prod status=500 | stats count

Result: 1

Results are coming from below sample logs:

::ffff:10.244.3.38 - - [06/Aug/2020:20:14:03 +0000] "GET /api/v1/workspace/getEngagement2?id=123 HTTP/1.1" 500 39 "https://atlas.intenal.noman.com" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.105 Safari/537.36"

I have defined a Field type: status for the above which uses Inline Field Extraction: ^[^"\n]*"(?P<method>\w+)[^"\n]*"\s+(?P<status>\d+)

Now when I perform a new search

index=logs_glbl sourcetype=kube:container:app-name namespace=prod | stats count by status

I don’t get the status 500 error. My results exclude the 500 status. It is also probably missing other http statuses too.

status	count
200	515
302	152
304	8
401	71
409	7

Re: stats count by generating inconsistent result

oscar84x — Fri, 07 Aug 2020 19:08:47 GMT

Very simple suggestion, but are you certain that the events within the time window you're using contain status=500?

Also, out of curiosity, I'm not familiar with what the "P" (?P<status>\d+) in your name capture represents.

Re: stats count by generating inconsistent result

noman377 — Fri, 07 Aug 2020 19:35:04 GMT

@oscar84x :: Yes. Within the same time frame (e.g., Last 24 hours, Last 7 days), I'm seeing search results that are not consistent. However, The alerts I receive based on "status" is accurate. To extract the http status, like 200, 500 etc., I used the regular expression to create the "status" field extraction.
Since, "| stats count by status" does not bring the 500 statuses, my dashboard is off not much use.