Knowledge Management

how to remove duplicate event when I am using data model acceleration

hqw
Path Finder

Hi all,

I am using data model acceleration method to do the dashboard acceleration. However, I find duplicate events in splunk. I know if use search, we can use dedup command. However, for data model root events, it doesn't allowed and "|" , for example, my original constrain is "index=123", now i want to remove those duplicate events before building data model, but splunk doesn't allow me write :"index=123 |dedup _row". Besides, from my dashboard panels, all panels created from data model must started with "|pivot data model name", and I also can't add any dedup command. Can anyone help me on this?

Thanks a lot

0 Karma

oajengui
Explorer

I'm facing the same situation where i found duplicates in my datamodel, because the dataset I created for the model is root event based and I have duplicates in my indexed events , and I couldn't find any command to de-duplicate the data from the model, so a workaround for that is to create a dataset for the data model based on root search instead of root event, and in that search add a dedup command, that way the data in the data model should not have duplicates, but it would be easier if there was a command that be used after creating the data model to deduplicate the data, because in my case i had to recreate the dataset of my datamodel

0 Karma

helge
Builder

The correct answer depends on the exact nature of your data, but you might be able to get rid of duplicate events by defining a set of fields that combined ensure events are unique and then adding splitrow statements for each of the fields, e.g.

| pivot datamodel object
   first(someField)
   splitrow field1
   splitrow field2
   splitrow field3
0 Karma
Get Updates on the Splunk Community!

Optimize Cloud Monitoring

  TECH TALKS Optimize Cloud Monitoring Tuesday, August 13, 2024  |  11:00AM–12:00PM PST   Register to ...

What's New in Splunk Cloud Platform 9.2.2403?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.2.2403! Analysts can ...

Stay Connected: Your Guide to July and August Tech Talks, Office Hours, and Webinars!

Dive into our sizzling summer lineup for July and August Community Office Hours and Tech Talks. Scroll down to ...