Splunk Search

Identical searches with different results because of dedup command

tsomod
Path Finder

Hi! I have two identical searches running on the same search head but with different time frames. What confuses me is that where the searches overlap in time, the results are different from one to the other, which doesn't make much sense to me. The two searches are:

index=XXXXXXXXXXXX sourcetype=XXXXXXXXXXX earliest=0 latest=@h | dedup src_ip sortby +_time | table src_ip,_time

and

index=XXXXXXXXXXXX sourcetype=XXXXXXXXXXX  earliest=-1h@h latest=@h | dedup src_ip sortby +_time | table src_ip,_time

As you can see the searches are identical except for the time frames. When I run the second search it results in MORE events over the last hour of the search than the first search over the same last hour. The searches are run at the same time. Any ideas why this happens?

Tags (2)
0 Karma
1 Solution

tsomod
Path Finder

After a few hours of inspection I actually figured out what was wrong. It's the dedup command. Because the command is used on a longer timespan in the first search than in the second it is only natural that some src_ip's might be found in the first search that are not present in the second. Because of this fewer src_ip's are deduped in the second search, resulting in more events being output!

View solution in original post

0 Karma

tsomod
Path Finder

After a few hours of inspection I actually figured out what was wrong. It's the dedup command. Because the command is used on a longer timespan in the first search than in the second it is only natural that some src_ip's might be found in the first search that are not present in the second. Because of this fewer src_ip's are deduped in the second search, resulting in more events being output!

0 Karma

richgalloway
SplunkTrust
SplunkTrust

@tsomod, If your problem is resolved, please accept an answer (you can accept your own answer) to help future readers.

---
If this reply helps you, Karma would be appreciated.
0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi tsomod,
probably you're receiving logs in the meanwhile you're searching.
try to execute searches using past and fixed time frames (e.g. -1h@h @h and -2h@h -1h@h), in other words, don't use now as latest.
Bye.
Giuseppe

0 Karma

tsomod
Path Finder

After a few hours of inspection I actually figured out what was wrong. It's the dedup command. Because the command is used on a longer timespan in the first search than in the second it is only natural that some src_ip 's might be found in the first search that are not present in the second. Because of this fewer src_ip's are deduped in the second search, resulting in more events being output! Thank you, however, for your response!

0 Karma
Get Updates on the Splunk Community!

.conf24 | Day 0

Hello Splunk Community! My name is Chris, and I'm based in Canberra, Australia's capital, and I travelled for ...

Enhance Security Visibility with Splunk Enterprise Security 7.1 through Threat ...

 (view in My Videos)Struggling with alert fatigue, lack of context, and prioritization around security ...

Troubleshooting the OpenTelemetry Collector

  In this tech talk, you’ll learn how to troubleshoot the OpenTelemetry collector - from checking the ...