Identical searches with different results because of dedup command

[tsomod]

Hi! I have two identical searches running on the same search head but with different time frames. What confuses me is that where the searches overlap in time, the results are different from one to the other, which doesn't make much sense to me. The two searches are:

index=XXXXXXXXXXXX sourcetype=XXXXXXXXXXX earliest=0 latest=@h | dedup src_ip sortby +_time | table src_ip,_time

and

index=XXXXXXXXXXXX sourcetype=XXXXXXXXXXX  earliest=-1h@h latest=@h | dedup src_ip sortby +_time | table src_ip,_time

As you can see the searches are identical except for the time frames. When I run the second search it results in MORE events over the last hour of the search than the first search over the same last hour. The searches are run at the same time. Any ideas why this happens?

[tsomod · Accepted solution]

After a few hours of inspection I actually figured out what was wrong. It's the dedup command. Because the command is used on a longer timespan in the first search than in the second it is only natural that some src_ip's might be found in the first search that are not present in the second. Because of this fewer src_ip's are deduped in the second search, resulting in more events being output!

[richgalloway]

@tsomod, If your problem is resolved, please accept an answer (you can accept your own answer) to help future readers.

[gcusello]

Hi tsomod,
probably you're receiving logs in the meanwhile you're searching.
try to execute searches using past and fixed time frames (e.g. -1h@h @h and -2h@h -1h@h), in other words, don't use now as latest.
Bye.
Giuseppe

[tsomod]

After a few hours of inspection I actually figured out what was wrong. It's the dedup command. Because the command is used on a longer timespan in the first search than in the second it is only natural that some src_ip 's might be found in the first search that are not present in the second. Because of this fewer src_ip's are deduped in the second search, resulting in more events being output! Thank you, however, for your response!