I am using data model acceleration method to do the dashboard acceleration. However, I find duplicate events in splunk. I know if use search, we can use dedup command. However, for data model root events, it doesn't allowed and "|" , for example, my original constrain is "index=123", now i want to remove those duplicate events before building data model, but splunk doesn't allow me write :"index=123 |dedup _row". Besides, from my dashboard panels, all panels created from data model must started with "|pivot data model name", and I also can't add any dedup command. Can anyone help me on this?
Thanks a lot
The correct answer depends on the exact nature of your data, but you might be able to get rid of duplicate events by defining a set of fields that combined ensure events are unique and then adding splitrow statements for each of the fields, e.g.
| pivot datamodel object first(someField) splitrow field1 splitrow field2 splitrow field3
I'm facing the same situation where i found duplicates in my datamodel, because the dataset I created for the model is root event based and I have duplicates in my indexed events , and I couldn't find any command to de-duplicate the data from the model, so a workaround for that is to create a dataset for the data model based on root search instead of root event, and in that search add a dedup command, that way the data in the data model should not have duplicates, but it would be easier if there was a command that be used after creating the data model to deduplicate the data, because in my case i had to recreate the dataset of my datamodel