Splunk Search

Dedup is removing the duplicate fields which is having the Unique value in other column

LRathinakumar
Explorer

Hello Splunkers,

I have used a query in the search for mitre fields extraction and after the extraction i have got the results with the query name and the technique_id. But here the problem comes. Each query is having the technique_id and the sub technique_id, so i have matched the sub technique_id with the technique_id and the results is shown with the same rule name two times with the technique_id. So i want to remove  the duplicate rule name, if so i used the dedup then the rule having the other technique_id is also getting removed. I have attached the screenshot for reference...

The query i used for getting the results is 



| rest /services/configs/conf-analyticstories
| where annotations!=""
| spath input=annotations path=mitre_attack{} output=mitre_attack
| eval rule_name=ltrim(title,"savedsearch://")
| fields rule_name,mitre_attack
| join rule_name
[| rest /services/configs/conf-analyticstories
| where searches!=""
| eval rule_name=searches
| table title,rule_name
| eval rule_name=trim(rule_name,"[")
| eval rule_name=trim(rule_name,"]")
| eval rule_name=split(rule_name,",")
| mvexpand rule_name
| eval rule_name=trim(rule_name," ")
| eval rule_name=trim(rule_name,"\"")
]
| append
[| rest services/configs/conf-savedsearches
| eval rule_name=title
| search action.correlationsearch.annotations="*"
| spath input=action.correlationsearch.annotations path=mitre_attack{} output=mitre_attack
| fields rule_name, mitre_attack]
| eval technique_name = if(match(mitre_attack,"^T\d\d\d"),null(), mitre_attack)
| lookup mitre_tt_lookup technique_name OUTPUT technique_id as tmp_id0
| eval tmp_id1 = if(match(mitre_attack,"^T\d\d\d"), mitre_attack, null())
| eval technique_id=coalesce(tmp_id0, tmp_id1)
| where NOT isnull(technique_id)
| table rule_name, technique_id
| inputlookup mitre_user_rule_technique_lookup append=true
| inputlookup mitre_app_rule_technique_lookup append=true
| makemv tokenizer="([^\n\s]+)" technique_id
| mvexpand technique_id
| dedup rule_name,technique_id
| join rule_name
[| rest services/configs/conf-savedsearches
| eval rule_name=title
| eval stage= if(disabled == 1, "Disabled", "Enabled")
| table rule_name, stage
]
| eval subtechnique_id=if(match(technique_id,"\."),technique_id,null())
| eval technique_id=if(match(technique_id,"\."),replace(technique_id,"\.\d+",""),technique_id)
|search stage=Enabled
|table rule_name,technique_id

 

LRathinakumar_0-1676800735000.png

 

Thanks in advance....

0 Karma
1 Solution

ITWhisperer
SplunkTrust
SplunkTrust

Repeat the dedup line at the end

| dedup rule_name,technique_id 

Or don't edit the technique_id after the dedup to see the different versions

| eval technique_id=if(match(technique_id,"\."),replace(technique_id,"\.\d+",""),technique_id)

View solution in original post

ITWhisperer
SplunkTrust
SplunkTrust

Repeat the dedup line at the end

| dedup rule_name,technique_id 

Or don't edit the technique_id after the dedup to see the different versions

| eval technique_id=if(match(technique_id,"\."),replace(technique_id,"\.\d+",""),technique_id)
Get Updates on the Splunk Community!

Splunk Observability Cloud | Unified Identity - Now Available for Existing Splunk ...

Raise your hand if you’ve already forgotten your username or password when logging into an account. (We can’t ...

Index This | How many sides does a circle have?

February 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

Registration for Splunk University is Now Open!

Are you ready for an adventure in learning?   Brace yourselves because Splunk University is back, and it's ...