topic How to remove duplicates? in Splunk Search

How to remove duplicates?

wanda619 — Wed, 10 Aug 2022 14:12:43 GMT

Client	Error	Error Results	Error ResultsPrevious week	Percent of Total	PercentDifference
abc	1003	2	0	12.5	0
abc	1003	3		12.5	0
abc	1013	1	2	342	-50
abc	1027	3	3	5	0
abc	1027	5	xyz	43	zyz
abc	1013	2	zyz	432	et
abc	Total	16	zyds	423	tert

My code is --

| bucket _time span=1w | stats count as Result by LicenseKey, Error_Code | eval Client=coalesce(Client,LicenseKey) | eventstats sum(Result) as Total by Client | eval PercentOfTotal = round((Result/Total)*100,3) | sort - _time | streamstats current=f latest(Result) as Result_Prev by LicenseKey | eval PercentDifference = round(((Result/Result_Prev)-1)*100,2) | fillnull value="0" | append [ search index=abc sourcetype=yxx source= bff ErrorCode!=0 | `DedupDHI` | lookup abc LicenseKey OUTPUT Client | eval Client=coalesce(Client,LicenseKey) | stats count as Result by Client | eval ErrorCode="Total", PercentOfTotal=100] | lookup xyz_ErrorCodes ErrorCode OUTPUT Description | lookup uyz LicenseKey OUTPUT Client | eval Client=coalesce(Client,LicenseKey) | eval Error=if(ErrorCode!="Total", ErrorCode+" ("+coalesce(Description,"Description Missing - Update xyz_ErrorCodes")+")", ErrorCode) | fields Client, Error, Result, PercentOfTotal, PercentDifference, Error results previous week | sort CustomerName, Error, PercentDifference

Still not able to figure out the duplicate row issue, single row for one each error combined with total.

any suggestions please?

Re: How to remove Duplicates?

richgalloway — Tue, 09 Aug 2022 23:58:59 GMT

Please explain or provide a mock-up of the expected results.

Re: How to remove Duplicates?

bowesmana — Wed, 10 Aug 2022 00:11:41 GMT

I believe you already asked this

https://community.splunk.com/t5/Splunk-Search/How-to-remove-duplicates-from-the-table/m-p/608750

Do you want to remove all other fields in the table other than Error? What should occur with these two rows

abc	1027	3	3	5	0
abc	1027	5	xyz	43	zyz

for previous week and other columns?

Simple answer would be

| stats sum("Error Results") as "Error Results" values(*) as * by Client Error

but you need to a bit clearer about the removal part of duplicates.

The above would sum the error counts and just show all the values collected in the same row as the error.

I am assuming your sorting by CustomerName in your search is not what you meant/wanted, as there is no CustomerName field in your data