How to merge closest events by time for each host?

[IVV]

Hello!

I have a lot of different events of such kind:
host="A" datetime="time1.1" other_field="fld"
host="A" datetime="time1.2" other_field="fld"
host="A" datetime="time1.3" other_field="fld"
host="B" datetime="time2.1" other_field="fld"
host="B" datetime="time2.1" other_field="fld"
host="A" datetime="time3.1" other_field="fld"
host="A" datetime="time3.2" other_field="fld"
host="A" datetime="time4.1" other_field="fld"
host="A" datetime="time4.2" other_field="fld"
host="A" datetime="time4.3" other_field="fld"

And want to report it by email like this:
host="A" datetime="time1" other_field="fld"
host="B" datetime="time2" other_field="fld"
host="A" datetime="time3" other_field="fld"
host="A" datetime="time4" other_field="fld"

So I want to merge closest events (by time) for one host. What is the right way to do it?

Thanks!

[somesoni2 · Accepted solution]

Give this a try

your base search | bucket span=1s _time | dedup _time,host,other_field

[cpetterborg]

What is the difference in the time sets (1.1, 1.2, 1.3 or 3.1, 3.2, etc.)? Are you grouping them because they are within a certain range, or is there some other criteria that you are using?

[IVV]

It depends. But in fact it is a certain range. I mean there are several events on one host in one second, so time{1.1, 1.2, 1.3} are different milliseconds in one second. Or you can think about these values as sequence of time values in some time gap. I used transactions to merges events in this way

sourcetype='a_lot_of_events_in_little_time' other_field="fld" | transaction host maxspan=2s

But it is not the result I want, because transaction has multiple records inside itself. I need only one in time range.

Imagine one file modification creates a lot of events (for example 6 in one second) and sends them to logging system. But you interested only in the fact of file modification. So you want to merge these events into one and alert about it. But you don't want to miss next file modification that can happen right after the first one.

[tom_frotscher]

You should be able to use timechart for this. Is datetime equivalent to _time? Then you can do something like:

... | timechart span=2s earliest(_time) values(other_field) by host

Greetings

Tom

[IVV]

Thank you, but the result is not what I need.
Yes, datetime equivalent to _time. Here is an example: datetime="2014-12-04T16:23:29.705893Z".