topic Re: dedup events with same timestamp ? in Splunk Search

dedup events with same timestamp ?

pgadhari — Wed, 30 Sep 2020 02:45:56 GMT

I am facing issues wherein the events with same timestamp are not showing in results, when I dedup based on time, but I want all those events, even after dedup. Even epoch will be same for those events. Below is the sample query before dedup and result for the same.
Result are attached as an image. I want to show both the events in the results even after dedup, how can I achieve this ?

index=com vendor_action=comment_create|stats count by created_at,created_by_name|eval point=if(count>0,1,0) | eval epoch=strptime(created_at, "%Y-%m-%dT%H:%M:%S+%z")

Now, the query with dedup :

index=com vendor_action=comment_create|stats count by created_at,created_by_name|eval point=if(count>0,1,0) | eval epoch=strptime(created_at, "%Y-%m-%dT%H:%M:%S+%z") | dedup created_at

Re: dedup events with same timestamp ?

gcusello — Tue, 29 Oct 2019 11:37:09 GMT

Hi pgadhari,
sorry but I don't understand probably there's something I missed in translaction: if you want all the events why do you dedup?

Ciao.
Giuseppe

Re: dedup events with same timestamp ?

arjunpkishore5 — Tue, 29 Oct 2019 12:11:00 GMT

I guess you want to remove duplicate values and not entire rows. dedup removes rows based on the column specified. In your case, Instead of a dedup, you need this.

| stats values(*) as * by created_at

Let me know if this helps.

Cheers

Re: dedup events with same timestamp ?

pgadhari — Tue, 29 Oct 2019 12:20:21 GMT

Forgot to mention - Actually, the index which I mentioned is summary index, and in that I am getting duplicate events for every run. Its a saved search putting data into summary index and this search is scheduled search running every 5 minutes and getting data of last 15 minutes. Hence, I am getting duplicated events, hence I have to dedup. But doing dedup is removing one of the event of the same timestamps. Hope you got it ?

When I try to schedule it - to get data of last 5 minutes and running every 5 minutes - it is skipping some of the events, which is not helpful.

Re: dedup events with same timestamp ?

arjunpkishore5 — Wed, 30 Sep 2020 02:45:59 GMT

in your case, it looks like you should just change the key you're using to dedup, such as created_by_name. dedup returns one row per key

Re: dedup events with same timestamp ?

gaurav_maniar — Tue, 29 Oct 2019 12:32:59 GMT

Hi @pgadhari ,

I don't understand why you want to use dedup and also want to keep the events as well.
dedup created_at - it will remove all the events with same create_at value, irrespective of the other fields values.

In your case I would suggest try dedup _raw, it will only remove the events duplicate events, where the time and all other fields are same.
So in case for same created_at values, if event data is different, the query will return those events.

Accept & up-vote the answer if it helps.

Re: dedup events with same timestamp ?

gcusello — Tue, 29 Oct 2019 13:24:13 GMT

Hi pgadhari,
did you tryed to dedup for all the fields you have in Summary, or at least the more important, not only _time?

Ciao.
Giuseppe

Re: dedup events with same timestamp ?

pgadhari — Tue, 29 Oct 2019 16:01:10 GMT

I will try doing dedup with more than one field and check. I will revert on it.

Re: dedup events with same timestamp ?

pgadhari — Tue, 29 Oct 2019 16:07:45 GMT

No, I dont want to remove the dedup values, instead I want to keep it. As it is a summary index, it is generating duplicate events and thats why I am using "dedup created_at", but because of this dedup, the events which have same timestamp - either one of them is getting removed from the result, due to which I cannot see that user in our statistics. Hope you got it ?

Re: dedup events with same timestamp ?

pgadhari — Tue, 29 Oct 2019 16:09:50 GMT

I can try doing dedup the _raw events, but I am not sure, how it can help ? But see my above reply https://answers.splunk.com/comments/779814/view.html.

I dont want to remove those events, I want to keep it.

Re: dedup events with same timestamp ?

pgadhari — Wed, 30 Oct 2019 11:29:09 GMT

@arjunpkishore5 - I think after adding above query, its working. After adding above query, I did mvexpand by other field name and seems to be working. I need to monitor it for sometime. Once, its ok, I will accept this answer. Thanks.