How to club the messages to remove duplicates

[aditsss]

Hi Everyone,

I have one panel which consists of data like below:

_raw                                                                                                                                                                                             host

2021-03-04 04:27:13,219 INFO [Server-296] on.c.s.StandardProcessScheduler                     abc.phx.xcp.com

Disabling StandardControllerServiceNode versionedComponentId=null,                                             processGroup=StandardProcessGroup     

2021-03-04 04:27:13,219 INFO [Server-296] on.c.s.StandardProcessScheduler                     abc.phx.vpp.com

Disabling StandardControllerServiceNode versionedComponentId=null,                                             processGroup=StandardProcessGroup   

The issue I am facing is I want to remove the duplicates on basis of host.

I used dedup but all are removed .

But its not giving me correct value.

Can some one guide me how can I remove duplicates .

Below is my  query:

<query>index=abc sourcetype=xyz source="app.log" info $process_tok1$
| rex field=_raw "(?&lt;id&gt;[A_Za-z0-9]{8}[\-][A_Za-z0-9]{4}[\-][A_Za-z0-9]{4}[\-][A_Za-z0-9]{4}[\-][A_Za-z0-9]{12})"
| join type=outer id [inputlookup nifi_api_parent_e1.csv]|search $ckey$|eval ClickHere=url|rex field=url mode=sed "s/\\/\\//\\//g s/https:/https:\\//g"|dedup host
| table _time _raw host id parent_chain url
</query>

[ITWhisperer]

Are you sure you want dedup host? Try dedup _raw

[aditsss]

@ITWhisperer 

dedup _raw is not working. I am still getting all the logs

Below is my query:

index=abc sourcetype=xyz source="app.log" info $process_tok1$
| rex field=_raw "(?&lt;id&gt;[A_Za-z0-9]{8}[\-][A_Za-z0-9]{4}[\-][A_Za-z0-9]{4}[\-][A_Za-z0-9]{4}[\-][A_Za-z0-9]{12})"
| join type=outer id [inputlookup nifi_api_parent_e1.csv]|search $ckey$|eval ClickHere=url|rex field=url mode=sed "s/\\/\\//\\//g s/https:/https:\\//g"|dedup _raw
| table _time _raw host id parent_chain url

Can you guide me

[ITWhisperer]

Try dedup earlier in the search

index=abc sourcetype=xyz source="app.log" info $process_tok1$
| dedup _raw
| rex field=_raw "(?<id>[A_Za-z0-9]{8}[\-][A_Za-z0-9]{4}[\-][A_Za-z0-9]{4}[\-][A_Za-z0-9]{4}[\-][A_Za-z0-9]{12})"
| join type=outer id [inputlookup nifi_api_parent_e1.csv]|search $ckey$|eval ClickHere=url|rex field=url mode=sed "s/\\/\\//\\//g s/https:/https:\\//g"
| table _time _raw host id parent_chain url

[aditsss]

@ITWhisperer 

I tried still getting all:

index=abc sourcetype=xyz source="/var/log/nifi/nifi-app.log" info $process_tok1$|dedup _raw
| rex field=_raw "(?&lt;id&gt;[A_Za-z0-9]{8}[\-][A_Za-z0-9]{4}[\-][A_Za-z0-9]{4}[\-][A_Za-z0-9]{4}[\-][A_Za-z0-9]{12})"
| join type=outer id [inputlookup nifi_api_parent_e1.csv]|search $ckey$|eval ClickHere=url|rex field=url mode=sed "s/\\/\\//\\//g s/https:/https:\\//g"
| table _time _raw host id parent_chain url

[ITWhisperer]

The _raw records look the same - could there be trailing spaces? Try trim(_raw) before the dedup?

[aditsss]

@ITWhisperer 

Is my query correct for trim:

Its not taking the trim

index=abc sourcetype=xyz source="/var/log/nifi/nifi-app.log" info 668c47ec-e848-1d32-ffff-ffff96793d58|trim(_raw) |dedup _raw
| rex field=_raw "(?<id>[A_Za-z0-9]{8}[\-][A_Za-z0-9]{4}[\-][A_Za-z0-9]{4}[\-][A_Za-z0-9]{4}[\-][A_Za-z0-9]{12})"
| join type=outer id [inputlookup nifi_api_parent_e1.csv]|search |eval ClickHere=url|rex field=url mode=sed "s/\\/\\//\\//g s/https:/https:\\//g"
| table _time _raw host id parent_chain url

[ITWhisperer · Accepted solution]

eval _raw=trim(_raw)