https://community.splunk.com/t5/Splunk-Search/How-to-remove-consecutive-duplicate-values-by-host-from-custom/m-p/202369#M58697 <P>Assuming you have field <STRONG>host</STRONG> (which has host values) and field <STRONG>Value</STRONG> (which has Servicebus/ResponseTime) , please try this:</P> <PRE><CODE>index=perfmon collection=ServiceBus counter="Response Time" instance="Service" | sort host | fields host, Value | dedup host, Value consecutive=true | stats avg(Value) by host </CODE></PRE> Wed, 02 Nov 2016 06:06:46 GMT gokadroid 2016-11-02T06:06:46Z https://community.splunk.com/t5/Splunk-Search/How-to-remove-consecutive-duplicate-values-by-host-from-custom/m-p/202368#M58696 <P>Hi There, </P> <P>I am trying to figure out how to remove duplicates in a custom perfmon counters data that is exported to Splunk. </P> <P>We have a custom perfmon counter implemented for a service, which reports response time of a service when ever it is called. Otherwise, it will be returning the same old data from the last service response until the new request is served. The same service is hosted on multiple hosts and all the host report perfmon data. </P> <P>I am trying to find the average by removing all the consecutive duplicates by host and get an average. </P> <PRE><CODE>index=perfmon collection=ServiceBus counter="Response Time" instance="Service" | dedup Value host consecutive=true | chart avg(Value) </CODE></PRE> <P>This search is not accurate. </P> <P>Example Data :</P> <PRE><CODE>Host 1: Time Servicebus/ResponseTime 00:00:01 460 00:00:02 460 00:00:03 520 00:00:04 520 00:00:05 630 </CODE></PRE> <P>Host 2: </P> <PRE><CODE>Time Servicebus/ResponseTime 00:00:01 480 00:00:02 480 00:00:03 590 00:00:04 590 00:00:05 610 </CODE></PRE> <P>Since we are sending data to Splunk every 1s from all the hosts, this is how the data in the Splunk is represented (as per my knowledge). See below. In this case, the above search is not removing the duplicates. </P> <PRE><CODE>Time Servicebus/ResponseTime Host 00:00:01 460 Host 1 00:00:01 480 Host 2 00:00:02 460 Host 1 00:00:02 480 Host 2 00:00:03 520 Host 1 00:00:03 590 Host 2 00:00:04 520 Host 1 00:00:04 590 Host 2 00:00:05 630 Host 1 00:00:05 610 Host 2 </CODE></PRE> <P>Can someone help me on how to remove consecutive duplicates by host? </P> <P>Thanks,<BR /> Dayananda</P> Tue, 01 Nov 2016 22:31:08 GMT https://community.splunk.com/t5/Splunk-Search/How-to-remove-consecutive-duplicate-values-by-host-from-custom/m-p/202368#M58696 dayananda7449 2016-11-01T22:31:08Z https://community.splunk.com/t5/Splunk-Search/How-to-remove-consecutive-duplicate-values-by-host-from-custom/m-p/202369#M58697 <P>Assuming you have field <STRONG>host</STRONG> (which has host values) and field <STRONG>Value</STRONG> (which has Servicebus/ResponseTime) , please try this:</P> <PRE><CODE>index=perfmon collection=ServiceBus counter="Response Time" instance="Service" | sort host | fields host, Value | dedup host, Value consecutive=true | stats avg(Value) by host </CODE></PRE> Wed, 02 Nov 2016 06:06:46 GMT https://community.splunk.com/t5/Splunk-Search/How-to-remove-consecutive-duplicate-values-by-host-from-custom/m-p/202369#M58697 gokadroid 2016-11-02T06:06:46Z