stats count returns different results for indexed field

[bschaap]

I'm having difficulty understanding why Query 2 is returning a different count than the other two queries. The cluster_name field is an indexed field. Does anyone know why this is occurring and have any suggestions to resolve or troubleshoot this issue?

Query 1
index=infra_k8s* sourcetype="kube:objects:pods" | stats count by cluster_name
cluster_name count
...
clusterxyz 436
...

Query 2
index=infra_k8s* sourcetype="kube:objects:pods" cluster_name=clusterxyz | stats count by cluster_name
cluster_name count
clusterxyz 68

Query 3
index=infra_k8s* sourcetype="kube:objects:pods" | eval cluster=cluster_name | search cluster="clusterxyz" | stats count by cluster
cluster count
clusterxyz 436

[DalJeanis]

Okay, so there are two different relevant possibilities.

Run your other queries and something like this across the same time frame.

| tstats count where index=infra_k8s* sourcetype="kube:objects:pods" by cluster_name

That will give you the counts by solely the cluster_name fields that were established at index time. That should match your 68.

Any more than those, are from fields that are being extracted at search time.

(I'm assuming that devonprem in query3 is your actual clustername, or that every event in cluster devonprem is coincidentally having clusterxyz extracted also.)

[bschaap]

I was trying to make my queries generic but must have skipped over that one. I fixed the 3rd query of my original post.

I just tried tstats and it returned a count of 436 rather than 68.