Splunk Search

Dedup lost data, a bug?

april_tao
New Member

For below search :

eventtype=MYTYPE [search eventtype=MYTYPE | sort 0 _time desc | dedup fieldX | return 1000 source]

Expect to return the latest source for 1 fieldX value.

In our data, we have over 10,000,000 events for the latest source with fieldX=A, fieldX=B, fieldX=C respectively.
Thus expect the search returns over 30,000,000 results.
However, it returns the results for fieldX=A only.

Question : is the search correctly written? If yes, is this a bug of dedup? Is there any limitation of dedup about the result size? If we use a smaller dataset, dedup works properly with the same search.

Tags (2)
0 Karma

lguinn2
Legend

The problem is that the subsearch has a limit - and I don't see that you need the subsearch at all. You also do not need the sort, Splunk returns events in reverse time order (newest first) by default.Try this

eventtype=MYTYPE | dedup fieldX

This will return the most recent event for each value of fieldX.

0 Karma
Get Updates on the Splunk Community!

.conf24 | Day 0

Hello Splunk Community! My name is Chris, and I'm based in Canberra, Australia's capital, and I travelled for ...

Enhance Security Visibility with Splunk Enterprise Security 7.1 through Threat ...

(view in My Videos)Struggling with alert fatigue, lack of context, and prioritization around security ...

Troubleshooting the OpenTelemetry Collector

  In this tech talk, you’ll learn how to troubleshoot the OpenTelemetry collector - from checking the ...