Dedup is removing the duplicate fields which is having the Unique value in other column

LRathinakumar — Sun, 19 Feb 2023 10:01:53 GMT

Hello Splunkers,

I have used a query in the search for mitre fields extraction and after the extraction i have got the results with the query name and the technique_id. But here the problem comes. Each query is having the technique_id and the sub technique_id, so i have matched the sub technique_id with the technique_id and the results is shown with the same rule name two times with the technique_id. So i want to remove the duplicate rule name, if so i used the dedup then the rule having the other technique_id is also getting removed. I have attached the screenshot for reference...

The query i used for getting the results is



| rest /services/configs/conf-analyticstories 
| where annotations!="" 
| spath input=annotations path=mitre_attack{} output=mitre_attack 
| eval rule_name=ltrim(title,"savedsearch://") 
| fields rule_name,mitre_attack 
| join rule_name 
[| rest /services/configs/conf-analyticstories 
| where searches!="" 
| eval rule_name=searches 
| table title,rule_name 
| eval rule_name=trim(rule_name,"[") 
| eval rule_name=trim(rule_name,"]") 
| eval rule_name=split(rule_name,",") 
| mvexpand rule_name 
| eval rule_name=trim(rule_name," ") 
| eval rule_name=trim(rule_name,"\"")
] 
| append 
[| rest services/configs/conf-savedsearches 
| eval rule_name=title 
| search action.correlationsearch.annotations="*" 
| spath input=action.correlationsearch.annotations path=mitre_attack{} output=mitre_attack 
| fields rule_name, mitre_attack] 
| eval technique_name = if(match(mitre_attack,"^T\d\d\d"),null(), mitre_attack)
| lookup mitre_tt_lookup technique_name OUTPUT technique_id as tmp_id0
| eval tmp_id1 = if(match(mitre_attack,"^T\d\d\d"), mitre_attack, null())
| eval technique_id=coalesce(tmp_id0, tmp_id1)
| where NOT isnull(technique_id) 
| table rule_name, technique_id 
| inputlookup mitre_user_rule_technique_lookup append=true 
| inputlookup mitre_app_rule_technique_lookup append=true 
| makemv tokenizer="([^\n\s]+)" technique_id 
| mvexpand technique_id 
| dedup rule_name,technique_id 
| join rule_name 
[| rest services/configs/conf-savedsearches 
| eval rule_name=title 
| eval stage= if(disabled == 1, "Disabled", "Enabled")
| table rule_name, stage
] 
| eval subtechnique_id=if(match(technique_id,"\."),technique_id,null())
| eval technique_id=if(match(technique_id,"\."),replace(technique_id,"\.\d+",""),technique_id)
|search stage=Enabled
|table rule_name,technique_id

Thanks in advance....

Re: Dedup is removing the duplicate fields which is having the Unique value in other column

ITWhisperer — Sun, 19 Feb 2023 10:37:35 GMT

Repeat the dedup line at the end

| dedup rule_name,technique_id

Or don't edit the technique_id after the dedup to see the different versions

| eval technique_id=if(match(technique_id,"\."),replace(technique_id,"\.\d+",""),technique_id)

topic Dedup is removing the duplicate fields which is having the Unique value in other column in Splunk Search

Dedup is removing the duplicate fields which is having the Unique value in other column

Re: Dedup is removing the duplicate fields which is having the Unique value in other column