Hi Everyone,
I have one panel which consists of data like below:
_raw host
2021-03-04 04:27:13,219 INFO [Server-296] on.c.s.StandardProcessScheduler abc.phx.xcp.com
Disabling StandardControllerServiceNode versionedComponentId=null, processGroup=StandardProcessGroup
2021-03-04 04:27:13,219 INFO [Server-296] on.c.s.StandardProcessScheduler abc.phx.vpp.com
Disabling StandardControllerServiceNode versionedComponentId=null, processGroup=StandardProcessGroup
The issue I am facing is I want to remove the duplicates on basis of host.
I used dedup but all are removed .
But its not giving me correct value.
Can some one guide me how can I remove duplicates .
Below is my query:
<query>index=abc sourcetype=xyz source="app.log" info $process_tok1$
| rex field=_raw "(?<id>[A_Za-z0-9]{8}[\-][A_Za-z0-9]{4}[\-][A_Za-z0-9]{4}[\-][A_Za-z0-9]{4}[\-][A_Za-z0-9]{12})"
| join type=outer id [inputlookup nifi_api_parent_e1.csv]|search $ckey$|eval ClickHere=url|rex field=url mode=sed "s/\\/\\//\\//g s/https:/https:\\//g"|dedup host
| table _time _raw host id parent_chain url
</query>
Are you sure you want dedup host? Try dedup _raw
dedup _raw is not working. I am still getting all the logs
Below is my query:
index=abc sourcetype=xyz source="app.log" info $process_tok1$
| rex field=_raw "(?<id>[A_Za-z0-9]{8}[\-][A_Za-z0-9]{4}[\-][A_Za-z0-9]{4}[\-][A_Za-z0-9]{4}[\-][A_Za-z0-9]{12})"
| join type=outer id [inputlookup nifi_api_parent_e1.csv]|search $ckey$|eval ClickHere=url|rex field=url mode=sed "s/\\/\\//\\//g s/https:/https:\\//g"|dedup _raw
| table _time _raw host id parent_chain url
Can you guide me
Try dedup earlier in the search
index=abc sourcetype=xyz source="app.log" info $process_tok1$
| dedup _raw
| rex field=_raw "(?<id>[A_Za-z0-9]{8}[\-][A_Za-z0-9]{4}[\-][A_Za-z0-9]{4}[\-][A_Za-z0-9]{4}[\-][A_Za-z0-9]{12})"
| join type=outer id [inputlookup nifi_api_parent_e1.csv]|search $ckey$|eval ClickHere=url|rex field=url mode=sed "s/\\/\\//\\//g s/https:/https:\\//g"
| table _time _raw host id parent_chain url
I tried still getting all:
index=abc sourcetype=xyz source="/var/log/nifi/nifi-app.log" info $process_tok1$|dedup _raw
| rex field=_raw "(?<id>[A_Za-z0-9]{8}[\-][A_Za-z0-9]{4}[\-][A_Za-z0-9]{4}[\-][A_Za-z0-9]{4}[\-][A_Za-z0-9]{12})"
| join type=outer id [inputlookup nifi_api_parent_e1.csv]|search $ckey$|eval ClickHere=url|rex field=url mode=sed "s/\\/\\//\\//g s/https:/https:\\//g"
| table _time _raw host id parent_chain url
The _raw records look the same - could there be trailing spaces? Try trim(_raw) before the dedup?
Is my query correct for trim:
Its not taking the trim
index=abc sourcetype=xyz source="/var/log/nifi/nifi-app.log" info 668c47ec-e848-1d32-ffff-ffff96793d58|trim(_raw) |dedup _raw
| rex field=_raw "(?<id>[A_Za-z0-9]{8}[\-][A_Za-z0-9]{4}[\-][A_Za-z0-9]{4}[\-][A_Za-z0-9]{4}[\-][A_Za-z0-9]{12})"
| join type=outer id [inputlookup nifi_api_parent_e1.csv]|search |eval ClickHere=url|rex field=url mode=sed "s/\\/\\//\\//g s/https:/https:\\//g"
| table _time _raw host id parent_chain url
eval _raw=trim(_raw)