Splunk Search

stats count by generating inconsistent result

noman377
Explorer

I have a very simple search:

index=logs_glbl sourcetype=kube:container:app-name namespace=prod status=500 | stats count

Result: 1

Results are coming from below sample logs:

::ffff:10.244.3.38 - - [06/Aug/2020:20:14:03 +0000] "GET /api/v1/workspace/getEngagement2?id=123 HTTP/1.1" 500 39 "https://atlas.intenal.noman.com" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.105 Safari/537.36"

I have defined a Field type: status for the above which uses Inline Field Extraction:  ^[^"\n]*"(?P<method>\w+)[^"\n]*"\s+(?P<status>\d+)

Now when I perform a new search

index=logs_glbl sourcetype=kube:container:app-name namespace=prod | stats count by status

I don’t get the status 500 error. My results exclude the 500 status. It is also probably missing other http statuses too.

status

count

200

515

302

152

304

8

401

71

409

7

Labels (3)
0 Karma

oscar84x
Contributor

Very simple suggestion, but are you certain that the events within the time window you're using contain status=500? 

Also, out of curiosity, I'm not familiar with what the "P" (?P<status>\d+) in your name capture represents. 

0 Karma

noman377
Explorer

@oscar84x :: Yes. Within the same time frame (e.g., Last 24 hours, Last 7 days), I'm seeing search results that are not consistent. However, The alerts I receive based on "status" is accurate. To extract the http status, like 200, 500 etc., I used the regular expression to create the "status" field extraction. 
Since, "| stats count by status" does not bring the 500 statuses, my dashboard is off not much use.

0 Karma
Got questions? Get answers!

Join the Splunk Community Slack to learn, troubleshoot, and make connections with fellow Splunk practitioners in real time!

Meet up IRL or virtually!

Join Splunk User Groups to connect and learn in-person by region or remotely by topic or industry.

Get Updates on the Splunk Community!

Laser Bananas and Edge Hubs: Exploring Operational Technology (OT) Data Through a ...

  OT is a different environment to traditional IT and can have interesting challenges when interfacing the ...

Event Series: Mastering AI Tokenomics and Splunk Agent Observability

Beyond the Black Box: Correlating AI Performance and Tokenomics with Splunk Agent Observability   As ...

span_metrics: The OpenTelemetry-Idiomatic Way to See Inside Your Services

You open a trace in Splunk Observability Cloud and everything looks fine. One root span, order-pipeline, with ...