This is perfect. Thank you! Only had to add the missing "by" in  | eventstats values(pod_name_all) as pod_name_all importance index=abc sourcetype=kubectl | lookup pod_list pod_name_lookup as pod_name OUTPUT pod_name_lookup | where sourcetype == "kubectl" | bin span=1h@h _time | stats values(pod_name_lookup) as pod_name_lookup values(pod_name_all) as pod_name_all by importance _time | append [ inputlookup pod_list | rename pod_name_lookup as pod_name_all] | eventstats values(pod_name_all) as pod_name_all by importance | eval missing = if(isnull(pod_name_all), pod_name_all, mvappend(missing, mvmap(pod_name_all, if(pod_name_all IN (pod_name_lookup), null(), pod_name_all)))) | where isnotnull(missing) | timechart span=1m@m dc(missing) by importance ... View more

Was able to get this to give me one line for the non-critical pods total missing count over time. index=abc sourcetype=kubectl importance=non-critical | lookup pod_list pod_name_lookup as pod_name OUTPUT pod_name_lookup | append [inputlookup pod_list where importance = non-critical | rename pod_name_lookup as pod_name_all] | eventstats values(pod_name_all) as pod_name_all | where sourcetype == "kubectl" | timechart span=1h@h values(pod_name_lookup) as pod_name_lookup values(pod_name_all) as pod_name_all | eval missing = mvappend(missing, mvmap(pod_name_all, if(pod_name_all IN (pod_name_lookup), null(), pod_name_all))) | timechart span=1h@h count(missing) as non-critical-pods-missing   Working towards the goal of being able to get another line for critical. ... View more

In my environment, unless something catastrophic happens, there will always be data from critical and non-critical pods being ingested into this index/sourcetype. I dont have an environment to test the addition you suggested to account for a period of time where no "non-critical" pods reporting, but as soon as i do i will test your updated query. Removing the dedup from your original suggestion seems to have cleared up the weird issue i was seeing where i was getting anamolous results in the timechart.  This seems to be working now which is great. Huge thanks!! index=abc sourcetype=kubectl importance=non-critical | lookup pod_list pod_name_lookup as pod_name OUTPUT pod_name_lookup | append [inputlookup pod_list where importance = non-critical | rename pod_name_lookup as pod_name_all] | eventstats values(pod_name_all) as pod_name_all | where sourcetype == "kubectl" | timechart span=1h@h values(pod_name_lookup) as pod_name_lookup values(pod_name_all) as pod_name_all | eval missing = mvappend(missing, mvmap(pod_name_all, if(pod_name_all IN (pod_name_lookup), null(), pod_name_all))) | where isnotnull(missing) | timechart span=1h@h count by missing    Is there an easy way to instead of  having a individual line for each "missing" pod, to either have a single line with the total count of "non-critical" pods and possibly also have two lines for "critical" and "non-critical"? I guess im also looking to have a timechart summary of the total count of missing non-critical and critical pods. Hope that makes sense. ... View more

Some context: I have a scripted input that sends the output of kubectl get po -A --show-labels into the abc index and kubectl sourcetype. Each pod has the importance label with a value of either critical or non-critical. Some pods havent had that label applied as of yet, that is why I had the fill null importance lines in my original query. Agree that they can be ignored for this task. An event in the index would look something like this:  _time namespace pod_name importance etc.... The other fields are irrelevant to this search. This data gets ingested into splunk every minute. Each pod's name is unique (one container per pod) so i believe the lookup table wildcard pod groups will map to the individual pod. For example, there would only be one apache pod, not apache-2 and apache-12. I dont believe there are any concerns with the grouping. When I say that I know the data is missing, I mean that there is a pod (kafka) that hasnt been deployed for over a week. It is not being reported by the scripted input I mentioned earlier. That pod is listed in the pod lookup as (kafka-*) Unfortunately I am unable to provided screenshots. The trouble I am having with the query you generously shared is that when it runs and gets visualized as a line chart with the 24h time range picked, the kafka pod shows a count of 1(missing) 24 hours ago, then goes to 0 for the rest of time up until the current hour, where it returns to 1. Nothing changed during that time with the actual deployment status of that pod. ... View more

Thank you for this. I feel it is close but I am getting some inconsistent/incomplete findings in the search. I have a pod that I know is "missing" it is in the pod lookup table , but is not deployed. The Line chart shows that it is missing in the current hour but not in the previous hours that i know that it was missing.  ... View more

This is perfect. Thank you very much! ... View more

Thanks for the response. Im not getting any matches though. Everything is coming back as count=0 even though there are entries in the lookup that should match. ... View more

@yuanliu wrote: This ask could have two interpretations.  The simple one is extremely simple.  Let me give you the formula first.   | inputlookup pod_name_lookup where NOT [search index=abc sourcetype=kubectl | eval pod_name = mvindex(split(pod_name, "-"), 0) | stats values(pod_name) as pod_name] | stats dc(pod_name) as count values(pod_name) as pod_name by importance   This query gets me really close. The one edge case I did not bring up is that some pods have multiple parts of the expected name that are also split by dashes. For example, I would have this in the lookup: podd-unique-name critical   and need to match podd-unique-name-h98erg-n2439f Running critical from the results.   Yes the "importance" in both will match exactly, but it is only important in the lookup field. The goal of this is to display pods that are not found in the search results compared to the inputlookup, and using the "importance" field from the lookup display the missing pod name and importance. ... View more