Splunk Search

dedup events with same timestamp ?

pgadhari
Builder

I am facing issues wherein the events with same timestamp are not showing in results, when I dedup based on time, but I want all those events, even after dedup. Even epoch will be same for those events. Below is the sample query before dedup and result for the same.
Result are attached as an image. I want to show both the events in the results even after dedup, how can I achieve this ?

index=com vendor_action=comment_create|stats count by created_at,created_by_name|eval point=if(count>0,1,0) | eval epoch=strptime(created_at, "%Y-%m-%dT%H:%M:%S+%z")alt text

Now, the query with dedup :

index=com vendor_action=comment_create|stats count by created_at,created_by_name|eval point=if(count>0,1,0) | eval epoch=strptime(created_at, "%Y-%m-%dT%H:%M:%S+%z") | dedup created_at

alt text

0 Karma
1 Solution

arjunpkishore5
Motivator

I guess you want to remove duplicate values and not entire rows. dedup removes rows based on the column specified. In your case, Instead of a dedup, you need this.

| stats values(*) as * by created_at

Let me know if this helps.

Cheers

View solution in original post

0 Karma

gaurav_maniar
Builder

Hi @pgadhari ,

I don't understand why you want to use dedup and also want to keep the events as well.
dedup created_at - it will remove all the events with same create_at value, irrespective of the other fields values.

In your case I would suggest try dedup _raw, it will only remove the events duplicate events, where the time and all other fields are same.
So in case for same created_at values, if event data is different, the query will return those events.

Accept & up-vote the answer if it helps.

0 Karma

pgadhari
Builder

I can try doing dedup the _raw events, but I am not sure, how it can help ? But see my above reply https://answers.splunk.com/comments/779814/view.html.

I dont want to remove those events, I want to keep it.

0 Karma

arjunpkishore5
Motivator

in your case, it looks like you should just change the key you're using to dedup, such as created_by_name. dedup returns one row per key

0 Karma

arjunpkishore5
Motivator

I guess you want to remove duplicate values and not entire rows. dedup removes rows based on the column specified. In your case, Instead of a dedup, you need this.

| stats values(*) as * by created_at

Let me know if this helps.

Cheers

0 Karma

pgadhari
Builder

@arjunpkishore5 - I think after adding above query, its working. After adding above query, I did mvexpand by other field name and seems to be working. I need to monitor it for sometime. Once, its ok, I will accept this answer. Thanks.

0 Karma

pgadhari
Builder

No, I dont want to remove the dedup values, instead I want to keep it. As it is a summary index, it is generating duplicate events and thats why I am using "dedup created_at", but because of this dedup, the events which have same timestamp - either one of them is getting removed from the result, due to which I cannot see that user in our statistics. Hope you got it ?

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi pgadhari,
sorry but I don't understand probably there's something I missed in translaction: if you want all the events why do you dedup?

Ciao.
Giuseppe

0 Karma

pgadhari
Builder

Forgot to mention - Actually, the index which I mentioned is summary index, and in that I am getting duplicate events for every run. Its a saved search putting data into summary index and this search is scheduled search running every 5 minutes and getting data of last 15 minutes. Hence, I am getting duplicated events, hence I have to dedup. But doing dedup is removing one of the event of the same timestamps. Hope you got it ?

When I try to schedule it - to get data of last 5 minutes and running every 5 minutes - it is skipping some of the events, which is not helpful.

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi pgadhari,
did you tryed to dedup for all the fields you have in Summary, or at least the more important, not only _time?

Ciao.
Giuseppe

0 Karma

pgadhari
Builder

I will try doing dedup with more than one field and check. I will revert on it.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...