Post Processing Search as default

[shayhibah]

I have multiple logs with the same unique field.
for instance:

Time: 10:00:00 Log-id: 0x1212 Message: ABCD
Time: 10:00:11 Log-id: 0x5565 Message: EFG
Time: 12:00:00 Log-id: 0x1212 Message: HIJK
Time: 10:00:00 Log-id: 0x5565 Message: LMNO
Time: 16:05:00 Log-id: 0x1212 Message: PQR

I would like that for every query user does, before showing query results, to filter out the results and to return only the latest log (Time field) for each unique log (Log-id field).

I have many dashboards and search bars and I would like that this will happen automatically for each query.
Is it possible? If so, Can anyone help me with this?

[niketn]

@shayhibah, if your raw events are already sorted in reverse chronological with Time order and fields are extracted including Log_id, you just need dedup based on Log_id field.

<yourBaseSearch>
| dedup Log_id

If they are not you would need to create field extraction and sort results in reverse chonological order by Time. Refer to the following example with run anywhere search based on your sample data.

|  makeresults
|  eval data="Time: 10:00:00 Log-id: 0x1212 Message: ABCD;Time: 10:00:11 Log-id: 0x5565 Message: EFG;Time: 12:00:00 Log-id: 0x1212 Message: HIJK;Time: 10:00:00 Log-id: 0x5565 Message: LMNO;Time: 16:05:00 Log-id: 0x1212 Message: PQR"
|  makemv data delim=";"
|  mvexpand data
|  rename data as _raw
|  rex "Time: (?<Time>[^\s]+)\sLog-id: (?<Log_id>[^\s]+)\sMessage: (?<Message>.*)"
|  sort - Time
|  dedup Time

[shayhibah]

Hi niketnilay,

First - thank you for your response.

I do know the dedup command but - I want that it will happen automatically for each query or out-of-the-box dashboard.
I dont wanna change every dashboard query or tell the user to add it whenever he send query using search bar.

Is there any configuration file to define that each query, before return the results to the user, do "dedup" on the result?

[niketn]

Instead of calling a base search you can create a macro and within the macro you can also have dedup command.

[shayhibah]

May I ask for example? how to connect the macro to some sourcetype I've created?