Dedup is removing the duplicate fields which is having the Unique value in other column

[LRathinakumar]

Hello Splunkers,

I have used a query in the search for mitre fields extraction and after the extraction i have got the results with the query name and the technique_id. But here the problem comes. Each query is having the technique_id and the sub technique_id, so i have matched the sub technique_id with the technique_id and the results is shown with the same rule name two times with the technique_id. So i want to remove  the duplicate rule name, if so i used the dedup then the rule having the other technique_id is also getting removed. I have attached the screenshot for reference...

The query i used for getting the results is 

| rest /services/configs/conf-analyticstories 
| where annotations!="" 
| spath input=annotations path=mitre_attack{} output=mitre_attack 
| eval rule_name=ltrim(title,"savedsearch://") 
| fields rule_name,mitre_attack 
| join rule_name 
[| rest /services/configs/conf-analyticstories 
| where searches!="" 
| eval rule_name=searches 
| table title,rule_name 
| eval rule_name=trim(rule_name,"[") 
| eval rule_name=trim(rule_name,"]") 
| eval rule_name=split(rule_name,",") 
| mvexpand rule_name 
| eval rule_name=trim(rule_name," ") 
| eval rule_name=trim(rule_name,"\"")
] 
| append 
[| rest services/configs/conf-savedsearches 
| eval rule_name=title 
| search action.correlationsearch.annotations="*" 
| spath input=action.correlationsearch.annotations path=mitre_attack{} output=mitre_attack 
| fields rule_name, mitre_attack] 
| eval technique_name = if(match(mitre_attack,"^T\d\d\d"),null(), mitre_attack)
| lookup mitre_tt_lookup technique_name OUTPUT technique_id as tmp_id0
| eval tmp_id1 = if(match(mitre_attack,"^T\d\d\d"), mitre_attack, null())
| eval technique_id=coalesce(tmp_id0, tmp_id1)
| where NOT isnull(technique_id) 
| table rule_name, technique_id 
| inputlookup mitre_user_rule_technique_lookup append=true 
| inputlookup mitre_app_rule_technique_lookup append=true 
| makemv tokenizer="([^\n\s]+)" technique_id 
| mvexpand technique_id 
| dedup rule_name,technique_id 
| join rule_name 
[| rest services/configs/conf-savedsearches 
| eval rule_name=title 
| eval stage= if(disabled == 1, "Disabled", "Enabled")
| table rule_name, stage
] 
| eval subtechnique_id=if(match(technique_id,"\."),technique_id,null())
| eval technique_id=if(match(technique_id,"\."),replace(technique_id,"\.\d+",""),technique_id)
|search stage=Enabled
|table rule_name,technique_id

Thanks in advance....

[ITWhisperer · Accepted solution]

Repeat the dedup line at the end

| dedup rule_name,technique_id 

Or don't edit the technique_id after the dedup to see the different versions

| eval technique_id=if(match(technique_id,"\."),replace(technique_id,"\.\d+",""),technique_id)