Splunk Search

Counting Occurrences Based Off Multiple Fields

mhenrick
New Member

Hi All,

I'm currently trying to test my field structure for dedup, but aren't sure of what best practices are for doing so/how to code them all.

As of now, since I'm trying to delete a duplication error in my log system, I'm trying to count the amount of response where three fields are equal across the logs (not within the logs). If the count is above 2, it means that with those 3 fields of some set values I must be counting not only the original and the copy, but also some other log.

Do you know how I could potentially code this efficiently? Will I have to merge the 3 fields into one and then compare them with a count, or can I keep them separate?
Also, do any of you have any other ideas for how I could check dedup? I'm open to any and all suggestions.

Thanks so much for your help!

Tags (2)
0 Karma

lguinn2
Legend

Try this

yoursearchhere-across-all-sources
| eval combinedField = null()
| eval combinedField = coalesce(possibleFieldName1,possibleFieldName2,possibleFieldName3)
| stats count(combinedField) by source
| stats count values(source)

This will give you what you want for one field, I think. First, it creates a single field called combinedField, based on all the possible names that the field could have across the sources (possibleFieldName1, possibleFieldName2, etc.)
Then it counts the number of times that the field appears across the sources, and finally reduces that to the number of sources where the field appears. It displays the count of sources along with a list of the source names.

This works for one field. However, I can't see how you would do this for multiple fields simultaneously, especially if the field names can vary across sources. But perhaps I did not clearly understand the question...

0 Karma

mhenrick
New Member

Hi Iguinn,

I actually think this may work alone.
| stats count by a, b, c

Also, if I wanted to keep other values, I think I'd do it this way:
| stats values(d),count by a,b,c

Will this work?

0 Karma

mhenrick
New Member

Hi Iguinn,

Thanks for the help, I do need to compare 3 fields at once. I think the best way would be to do a count by the values in a mvfield I can create. I'm a bit confused about your code above because I thought coalesce just takes a number of fields and returns the first one that is not null, not necessarily combining them?

Would my code below count all logs that match fields a, b, and c? Also, do you know a better way of making the combinedfield?

| eval combinedfield = a.b.c
| stats count(combinedfield)

Thanks again for the help.

0 Karma
Get Updates on the Splunk Community!

What's New in Splunk Cloud Platform 9.2.2403?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.2.2403! Analysts can ...

Stay Connected: Your Guide to July and August Tech Talks, Office Hours, and Webinars!

Dive into our sizzling summer lineup for July and August Community Office Hours and Tech Talks. Scroll down to ...

Edge Processor Scaling, Energy & Manufacturing Use Cases, and More New Articles on ...

Splunk Lantern is a Splunk customer success center that provides advice from Splunk experts on valuable data ...