Let me first point out that you can only determine if a group of pods as denoted in pod_name_lookup is completely absent (missing), not any individual pod_name. As such, your "timechart" can only ha...
See more...
Let me first point out that you can only determine if a group of pods as denoted in pod_name_lookup is completely absent (missing), not any individual pod_name. As such, your "timechart" can only have values 1 and 0 for each missing pod_name_lookup. Second, I want to note that calculations to fill null importance values is irrelevant to the problem in hand, therefore I will ignore them. The way to think through a solution is as follows: You want to populate a field that contains all non-critical pod_name_lookup values in every event so you can compare with running ones in each time interval. (Hint: eventstats.) In other words, if you have these pods _time pod_name sourcetype 2024-05-08 01:42:10 apache-12 kubectl 2024-05-08 01:41:58 apache-2 kubectl 2024-05-08 01:41:46 kakfa-8 kubectl 2024-05-08 01:41:00 apache-13 kubectl 2024-05-08 01:40:52 someapp-6 kubectl 2024-05-08 01:39:40 grafana-backup-11 kubectl 2024-05-08 01:39:34 apache-4 kubectl 2024-05-08 01:39:32 kafka-6 kubectl 2024-05-08 01:39:26 someapp-2 kubectl 2024-05-08 01:38:16 apache-12 kubectl 2024-05-08 01:38:10 grafana-backup-6 kubectl and pod_list lookup contains the following importance namespace pod_name_lookup critical ns1 kafka-* critical ns1 apache-* non-critical ns2 grafana-backup-* non-critical ns2 someapp-* (As you can see, I added "someapp-*" because in your illustration, only one app is "non-critical". This makes data nontrivial.) You will want to produce an intermediate table like this (please ignore the time interval differences just focus on material fields): _time pod_name_lookup pod_name_all 2024-05-08 01:35:00 2024-05-08 01:36:00 apache-* grafana-backup-* grafana-backup-* someapp-* 2024-05-08 01:37:00 kafka-* someapp-* grafana-backup-* someapp-* 2024-05-08 01:38:00 apache-* grafana-backup-* grafana-backup-* someapp-* 2024-05-08 01:39:00 apache-* someapp-* grafana-backup-* someapp-* 2024-05-08 01:40:00 apache-* kakfa-* grafana-backup-* someapp-* (This illustration assumes that you are looking for missing pods in each calendar minute; I know this is ridiculous, but it is easier to emulate.) From this table, you can calculate which value(s) in pod_name_all is/are missing from pod_name_lookup. (Hint: mvmap can be an easy method.) In SPL, this thought process can be implemented as index=abc sourcetype=kubectl importance=non-critical
| lookup pod_list pod_name_lookup as pod_name OUTPUT pod_name_lookup
| dedup pod_name
| append
[inputlookup pod_list where importance = non-critical
| rename pod_name_lookup as pod_name_all]
| eventstats values(pod_name_all) as pod_name_all
| where sourcetype == "kubectl"
| timechart span=1h@h values(pod_name_lookup) as pod_name_lookup values(pod_name_all) as pod_name_all
| eval missing = mvappend(missing, mvmap(pod_name_all, if(pod_name_all IN (pod_name_lookup), null(), pod_name_all)))
| where isnotnull(missing)
| timechart span=1h@h count by missing In the above, I changed time bucket to 1h@h (as opposed to 1m@m used in illustrations). You need to change that to whatever suits your needs. Here is an emulation used to produce the above tables and this chart: | makeresults format=csv data="_time, pod_name
10,apache-12
22,apache-2
34,kakfa-8
80,apache-13
88,someapp-6
160,grafana-backup-11
166,apache-4
168,kafka-6
174,someapp-2
244,apache-12
250,grafana-backup-6"
| eval _time = now() - _time
| eval sourcetype = "kubectl", importance = "non-critical"
| eval pod_name_lookup = replace(pod_name, "\d+", "*")
``` the above emulates
index=abc sourcetype=kubectl importance=non-critical
| lookup pod_list pod_name_lookup as pod_name OUTPUT pod_name_lookup
| dedup pod_name
```
| append
[makeresults format=csv data="namespace, pod_name_lookup, importance
ns1, kafka-*, critical
ns1, apache-*, critical
ns2, grafana-backup-*, non-critical
ns2, someapp-*, non-critical"
| where importance = "non-critical"
``` subsearch thus far emulates
| inputlookup pod_list where importance = non-critical
```
| rename pod_name_lookup as pod_name_all]
| eventstats values(pod_name_all) as pod_name_all
| where sourcetype == "kubectl"
| timechart span=1m@m values(pod_name_lookup) as pod_name_lookup values(pod_name_all) as pod_name_all
| eval missing = mvappend(missing, mvmap(pod_name_all, if(pod_name_all IN (pod_name_lookup), null(), pod_name_all)))
| where isnotnull(missing)
| timechart span=1m@m count by missing