Dashboards & Visualizations

How to club the messages to remove duplicates

aditsss
Builder

Hi Everyone,

I have one panel which consists of data like below:

_raw                                                                                                                                                                                             host

2021-03-04 04:27:13,219 INFO [Server-296] on.c.s.StandardProcessScheduler                     abc.phx.xcp.com

Disabling StandardControllerServiceNode versionedComponentId=null,                                             processGroup=StandardProcessGroup     

 

2021-03-04 04:27:13,219 INFO [Server-296] on.c.s.StandardProcessScheduler                     abc.phx.vpp.com

Disabling StandardControllerServiceNode versionedComponentId=null,                                             processGroup=StandardProcessGroup   

 

The issue I am facing is I want to remove the duplicates on basis of host.

I used dedup but all are removed .

But its not giving me correct value.

Can some one guide me how can I remove duplicates .

Below is my  query:

<query>index=abc sourcetype=xyz source="app.log" info $process_tok1$
| rex field=_raw "(?&lt;id&gt;[A_Za-z0-9]{8}[\-][A_Za-z0-9]{4}[\-][A_Za-z0-9]{4}[\-][A_Za-z0-9]{4}[\-][A_Za-z0-9]{12})"
| join type=outer id [inputlookup nifi_api_parent_e1.csv]|search $ckey$|eval ClickHere=url|rex field=url mode=sed "s/\\/\\//\\//g s/https:/https:\\//g"|dedup host
| table _time _raw host id parent_chain url
</query>

                                                

Labels (3)
0 Karma
1 Solution

ITWhisperer
Ultra Champion
eval _raw=trim(_raw)

View solution in original post

ITWhisperer
Ultra Champion

Are you sure you want dedup host? Try dedup _raw

0 Karma

aditsss
Builder

@ITWhisperer 

 

dedup _raw is not working. I am still getting all the logs

Below is my query:

index=abc sourcetype=xyz source="app.log" info $process_tok1$
| rex field=_raw "(?&lt;id&gt;[A_Za-z0-9]{8}[\-][A_Za-z0-9]{4}[\-][A_Za-z0-9]{4}[\-][A_Za-z0-9]{4}[\-][A_Za-z0-9]{12})"
| join type=outer id [inputlookup nifi_api_parent_e1.csv]|search $ckey$|eval ClickHere=url|rex field=url mode=sed "s/\\/\\//\\//g s/https:/https:\\//g"|dedup _raw
| table _time _raw host id parent_chain url

Can you guide me

0 Karma

ITWhisperer
Ultra Champion

Try dedup earlier in the search

index=abc sourcetype=xyz source="app.log" info $process_tok1$
| dedup _raw
| rex field=_raw "(?&lt;id&gt;[A_Za-z0-9]{8}[\-][A_Za-z0-9]{4}[\-][A_Za-z0-9]{4}[\-][A_Za-z0-9]{4}[\-][A_Za-z0-9]{12})"
| join type=outer id [inputlookup nifi_api_parent_e1.csv]|search $ckey$|eval ClickHere=url|rex field=url mode=sed "s/\\/\\//\\//g s/https:/https:\\//g"
| table _time _raw host id parent_chain url
0 Karma

aditsss
Builder

@ITWhisperer 

I tried still getting all:

index=abc sourcetype=xyz source="/var/log/nifi/nifi-app.log" info $process_tok1$|dedup _raw
| rex field=_raw "(?&lt;id&gt;[A_Za-z0-9]{8}[\-][A_Za-z0-9]{4}[\-][A_Za-z0-9]{4}[\-][A_Za-z0-9]{4}[\-][A_Za-z0-9]{12})"
| join type=outer id [inputlookup nifi_api_parent_e1.csv]|search $ckey$|eval ClickHere=url|rex field=url mode=sed "s/\\/\\//\\//g s/https:/https:\\//g"
| table _time _raw host id parent_chain url

Tags (1)
0 Karma

ITWhisperer
Ultra Champion

The _raw records look the same - could there be trailing spaces? Try trim(_raw) before the dedup?

0 Karma

aditsss
Builder

@ITWhisperer 

Is my query correct for trim:

Its not taking the trim

index=abc sourcetype=xyz source="/var/log/nifi/nifi-app.log" info 668c47ec-e848-1d32-ffff-ffff96793d58|trim(_raw) |dedup _raw
| rex field=_raw "(?<id>[A_Za-z0-9]{8}[\-][A_Za-z0-9]{4}[\-][A_Za-z0-9]{4}[\-][A_Za-z0-9]{4}[\-][A_Za-z0-9]{12})"
| join type=outer id [inputlookup nifi_api_parent_e1.csv]|search |eval ClickHere=url|rex field=url mode=sed "s/\\/\\//\\//g s/https:/https:\\//g"
| table _time _raw host id parent_chain url

0 Karma

ITWhisperer
Ultra Champion
eval _raw=trim(_raw)

View solution in original post