How to get value from two searches to match?

[ermanoj21yadav]

I have 2 logs. The first statement gets logged when a pod dies. The second gets logged when my app gets notified. Sometimes, the pod dies and my app doesn't get notified. I want to write an alert when the pod dies but my application doesn't get notified.

Log1 (when a pod dies):

index=log1 "Forced deletion of orphaned Pod" | rex "podnamespace/(?<machineName>(.*?))\s" 

Log2 (when my app gets notified):

index=conversation "*Clearing DMC pod" sourcetype="cui-orchestration-log" podname=<podNameWhichDied>

I tried several options, but I am unable to refer to the field 'machineName' created by rex in the Log1 inside Log2 even though machineName has the right pod name:

index=log1 "Forced deletion of orphaned Pod" | rex "podnamespace/(?<machineName>(.*?))\s" | stats count as podsCrashedCount by machineName| appendcols [search index=log2 "App is deleting pod" podname=$machineName| stats dc(podname) as deletedInApp] | where podsCrashedCount!=deletedInApp

[gcusello · Accepted solution]

Hi @ermanoj21yadav,

if you're sure to have from the second search less than 50,000 results you can use a subsearch like this:

index=log1 "Forced deletion of orphaned Pod" | rex "podnamespace/(?<machineName>(.*?))\s" NOT [ search index=log2 "App is deleting pod" podname=$machineName | dedup podname | rename podname AS deletedInApp | fields machineName ] 
| ...

in few words, in this woay you check if the machineNames from the second search aren't in the first search.

Remember that to use subsearches, you need to use the same fieldname (it's also case sensitive) in both main and sub searches.

If instead you aren't sure of the second search result number, you could use something like this:

(index=log1 "Forced deletion of orphaned Pod") OR [ search index=log2 "App is deleting pod" podname=$machineName | dedup podname | rename podname AS deletedInApp | fields machineName ] 
| rex "podnamespace/(?<machineName>(.*?))\s" 
| stats dc(index) AS dc_index values(index) AS index BY machineName 
| where dc_index =1 AND index=log1
| table machineName 

If you need more fields in the aoutput, you have to add them to the stats command as values.

Ciao.

Giuseppe

[ermanoj21yadav]

Exactly what I needed. Thanks Giuseppe !!

[gcusello]

Hi @ermanoj21yadav,

you're always welcome, see next time!

Ciao and happy splunking

Giuseppe