Counting Occurrences Based Off Multiple Fields

[mhenrick]

Hi All,

I'm currently trying to test my field structure for dedup, but aren't sure of what best practices are for doing so/how to code them all.

As of now, since I'm trying to delete a duplication error in my log system, I'm trying to count the amount of response where three fields are equal across the logs (not within the logs). If the count is above 2, it means that with those 3 fields of some set values I must be counting not only the original and the copy, but also some other log.

Do you know how I could potentially code this efficiently? Will I have to merge the 3 fields into one and then compare them with a count, or can I keep them separate?
Also, do any of you have any other ideas for how I could check dedup? I'm open to any and all suggestions.

Thanks so much for your help!

[lguinn2]

Try this

yoursearchhere-across-all-sources
| eval combinedField = null()
| eval combinedField = coalesce(possibleFieldName1,possibleFieldName2,possibleFieldName3)
| stats count(combinedField) by source
| stats count values(source)

This will give you what you want for one field, I think. First, it creates a single field called combinedField, based on all the possible names that the field could have across the sources (possibleFieldName1, possibleFieldName2, etc.)
Then it counts the number of times that the field appears across the sources, and finally reduces that to the number of sources where the field appears. It displays the count of sources along with a list of the source names.

This works for one field. However, I can't see how you would do this for multiple fields simultaneously, especially if the field names can vary across sources. But perhaps I did not clearly understand the question...

[mhenrick]

Hi Iguinn,

I actually think this may work alone.
| stats count by a, b, c

Also, if I wanted to keep other values, I think I'd do it this way:
| stats values(d),count by a,b,c

Will this work?

[mhenrick]

Hi Iguinn,

Thanks for the help, I do need to compare 3 fields at once. I think the best way would be to do a count by the values in a mvfield I can create. I'm a bit confused about your code above because I thought coalesce just takes a number of fields and returns the first one that is not null, not necessarily combining them?

Would my code below count all logs that match fields a, b, and c? Also, do you know a better way of making the combinedfield?

| eval combinedfield = a.b.c
| stats count(combinedfield)

Thanks again for the help.