Dashboards & Visualizations

How to club the messages to remove duplicates

aditsss
Motivator

Hi Everyone,

I have one panel which consists of data like below:

_raw                                                                                                                                                                                             host

2021-03-04 04:27:13,219 INFO [Server-296] on.c.s.StandardProcessScheduler                     abc.phx.xcp.com

Disabling StandardControllerServiceNode versionedComponentId=null,                                             processGroup=StandardProcessGroup     

 

2021-03-04 04:27:13,219 INFO [Server-296] on.c.s.StandardProcessScheduler                     abc.phx.vpp.com

Disabling StandardControllerServiceNode versionedComponentId=null,                                             processGroup=StandardProcessGroup   

 

The issue I am facing is I want to remove the duplicates on basis of host.

I used dedup but all are removed .

But its not giving me correct value.

Can some one guide me how can I remove duplicates .

Below is my  query:

<query>index=abc sourcetype=xyz source="app.log" info $process_tok1$
| rex field=_raw "(?&lt;id&gt;[A_Za-z0-9]{8}[\-][A_Za-z0-9]{4}[\-][A_Za-z0-9]{4}[\-][A_Za-z0-9]{4}[\-][A_Za-z0-9]{12})"
| join type=outer id [inputlookup nifi_api_parent_e1.csv]|search $ckey$|eval ClickHere=url|rex field=url mode=sed "s/\\/\\//\\//g s/https:/https:\\//g"|dedup host
| table _time _raw host id parent_chain url
</query>

                                                

Labels (3)
0 Karma
1 Solution

ITWhisperer
SplunkTrust
SplunkTrust
eval _raw=trim(_raw)

View solution in original post

ITWhisperer
SplunkTrust
SplunkTrust

Are you sure you want dedup host? Try dedup _raw

0 Karma

aditsss
Motivator

@ITWhisperer 

 

dedup _raw is not working. I am still getting all the logs

Below is my query:

index=abc sourcetype=xyz source="app.log" info $process_tok1$
| rex field=_raw "(?&lt;id&gt;[A_Za-z0-9]{8}[\-][A_Za-z0-9]{4}[\-][A_Za-z0-9]{4}[\-][A_Za-z0-9]{4}[\-][A_Za-z0-9]{12})"
| join type=outer id [inputlookup nifi_api_parent_e1.csv]|search $ckey$|eval ClickHere=url|rex field=url mode=sed "s/\\/\\//\\//g s/https:/https:\\//g"|dedup _raw
| table _time _raw host id parent_chain url

Can you guide me

0 Karma

ITWhisperer
SplunkTrust
SplunkTrust

Try dedup earlier in the search

index=abc sourcetype=xyz source="app.log" info $process_tok1$
| dedup _raw
| rex field=_raw "(?&lt;id&gt;[A_Za-z0-9]{8}[\-][A_Za-z0-9]{4}[\-][A_Za-z0-9]{4}[\-][A_Za-z0-9]{4}[\-][A_Za-z0-9]{12})"
| join type=outer id [inputlookup nifi_api_parent_e1.csv]|search $ckey$|eval ClickHere=url|rex field=url mode=sed "s/\\/\\//\\//g s/https:/https:\\//g"
| table _time _raw host id parent_chain url
0 Karma

aditsss
Motivator

@ITWhisperer 

I tried still getting all:

index=abc sourcetype=xyz source="/var/log/nifi/nifi-app.log" info $process_tok1$|dedup _raw
| rex field=_raw "(?&lt;id&gt;[A_Za-z0-9]{8}[\-][A_Za-z0-9]{4}[\-][A_Za-z0-9]{4}[\-][A_Za-z0-9]{4}[\-][A_Za-z0-9]{12})"
| join type=outer id [inputlookup nifi_api_parent_e1.csv]|search $ckey$|eval ClickHere=url|rex field=url mode=sed "s/\\/\\//\\//g s/https:/https:\\//g"
| table _time _raw host id parent_chain url

Tags (1)
0 Karma

ITWhisperer
SplunkTrust
SplunkTrust

The _raw records look the same - could there be trailing spaces? Try trim(_raw) before the dedup?

0 Karma

aditsss
Motivator

@ITWhisperer 

Is my query correct for trim:

Its not taking the trim

index=abc sourcetype=xyz source="/var/log/nifi/nifi-app.log" info 668c47ec-e848-1d32-ffff-ffff96793d58|trim(_raw) |dedup _raw
| rex field=_raw "(?<id>[A_Za-z0-9]{8}[\-][A_Za-z0-9]{4}[\-][A_Za-z0-9]{4}[\-][A_Za-z0-9]{4}[\-][A_Za-z0-9]{12})"
| join type=outer id [inputlookup nifi_api_parent_e1.csv]|search |eval ClickHere=url|rex field=url mode=sed "s/\\/\\//\\//g s/https:/https:\\//g"
| table _time _raw host id parent_chain url

0 Karma

ITWhisperer
SplunkTrust
SplunkTrust
eval _raw=trim(_raw)
Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

ICYMI - Check out the latest releases of Splunk Edge Processor

Splunk is pleased to announce the latest enhancements to Splunk Edge Processor.  HEC Receiver authorization ...

Introducing the 2024 SplunkTrust!

Hello, Splunk Community! We are beyond thrilled to announce our newest group of SplunkTrust members!  The ...