Knowledge Management

how to remove duplicate event when I am using data model acceleration

hqw
Path Finder

Hi all,

I am using data model acceleration method to do the dashboard acceleration. However, I find duplicate events in splunk. I know if use search, we can use dedup command. However, for data model root events, it doesn't allowed and "|" , for example, my original constrain is "index=123", now i want to remove those duplicate events before building data model, but splunk doesn't allow me write :"index=123 |dedup _row". Besides, from my dashboard panels, all panels created from data model must started with "|pivot data model name", and I also can't add any dedup command. Can anyone help me on this?

Thanks a lot

0 Karma

oajengui
Explorer

I'm facing the same situation where i found duplicates in my datamodel, because the dataset I created for the model is root event based and I have duplicates in my indexed events , and I couldn't find any command to de-duplicate the data from the model, so a workaround for that is to create a dataset for the data model based on root search instead of root event, and in that search add a dedup command, that way the data in the data model should not have duplicates, but it would be easier if there was a command that be used after creating the data model to deduplicate the data, because in my case i had to recreate the dataset of my datamodel

0 Karma

helge
Builder

The correct answer depends on the exact nature of your data, but you might be able to get rid of duplicate events by defining a set of fields that combined ensure events are unique and then adding splitrow statements for each of the fields, e.g.

| pivot datamodel object
   first(someField)
   splitrow field1
   splitrow field2
   splitrow field3
0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...