Splunk Search

Dedup is not working with mstats

sabari80
Explorer

We are streaming Dynatrace metric data into Splunk, for some reason we are seeing duplicate 'MessageDeduplicationId'. So trying to avoid the duplicate entries using dedup command. But not retrieving any results after using dedup command. Here is my initial query and getting results for this with duplicates-

| mstats sum(calc:service.thaa_stress_requests_count_lr_tags) As "Count" ,avg(calc:service.thaa_stress_requests_lr_tags) As "Response" where index=itsi_im_metrics AND source.name="DT_NonProd_SaaS" by Dimension.id | eval Response=round((Response/1000000),2), Count=round(Count,0) | search Dimension.id IN ("*Process.aspx") After adding dedup to avoid duplicate 'MessageDeduplicationId' , no results | mstats sum(calc:service.thaa_stress_requests_count_lr_tags) As "Count" ,avg(calc:service.thaa_stress_requests_lr_tags) As "Response" where index=itsi_im_metrics AND source.name="DT_NonProd_SaaS" by Dimension.id 
| eval Response=round((Response/1000000),2), Count=round(Count,0) 
| search Dimension.id IN ("*Process.aspx") 
| dedup MessageDeduplicationId sample payload: Dimension.id: xxxProcess.aspx Dimension.name: Literal Not Found MessageDeduplicationId: a901b712889217fc194cd0446a70325e aggregation: avg entity.service.id: xxx entity.service.name:xxxx metric_name:calc:service.thaa_stress_requests_lr_tags: 1613759 resolution: 1m source.name: xxxx unit: MicroSecond
Labels (2)
Tags (1)
0 Karma

sabari80
Explorer

sample Payload

=========

 

sample payload:

Dimension.id: xxxProcess.aspx

Dimension.name: Literal Not Found

MessageDeduplicationId: a901b712889217fc194cd0446a70325e

aggregation: avg

entity.service.id: xxx

entity.service.name:xxxx

metric_name:calc:xxxx_

lr_tags: 1613759

resolution: 1m s

ource.name: xxxx

unit: MicroSecond
0 Karma

sabari80
Explorer

Modified Query

==========

 

| mstats sum(calc:service.thaa_stress_requests_count_lr_tags) As "Count" ,avg(calc:service.thaa_stress_requests_lr_tags) As "Response" where index=itsi_im_metrics AND source.name="DT_NonProd_SaaS" by Dimension.id | eval Response=round((Response/1000000),2), Count=round(Count,0) | search Dimension.id IN ("*Process.aspx") | dedup MessageDeduplicationId 
0 Karma

sabari80
Explorer

Initial Query

==========

 

| mstats sum(calc:service.thaa_stress_requests_count_lr_tags) As "Count" ,avg(calc:service.thaa_stress_requests_lr_tags) As "Response" where index=itsi_im_metrics AND source.name="DT_NonProd_SaaS" by Dimension.id | eval Response=round((Response/1000000),2), Count=round(Count,0) | search Dimension.id IN ("*Process.aspx")
0 Karma
Get Updates on the Splunk Community!

Enhance Security Visibility with Splunk Enterprise Security 7.1 through Threat ...

(view in My Videos)Struggling with alert fatigue, lack of context, and prioritization around security ...

Troubleshooting the OpenTelemetry Collector

  In this tech talk, you’ll learn how to troubleshoot the OpenTelemetry collector - from checking the ...

Adoption of Infrastructure Monitoring at Splunk

  Splunk's Growth Engineering team showcases one of their first Splunk product adoption-Splunk Infrastructure ...