How to ignore a field during search so total count is correct

[colbymahan]

I have repeating error events that are identical except for a single id field value that is incremented for each occurrence. I want to have them be considered as the same, so i get an accurate total of occurrences of that error, rather than each one counted as a different Error message.

The scenario actually occurs in 2 ways. One is with the field value changing, and another is with a value in the actual error message changing. I assume the way to ignore it may be different for a Field vs a string in another field, so this may be a 2 part answer.

[lukejadamec]

Another way you can change the way mulitvalue fields are counted by breaking them down into single value fields. Here is an example where the field "Account" shows up twice in each event.

If you run the search:

index=main Account=* |stats count by Account

The sum of the Account stats will be twice the number of events, but you will see all values of the Account field.

If you run this search:

index=main Account=* |eval Account=(Account,0) |stats count by Account

The sum of the Account stats will be the same as the number of events, but you will only see the first value for the Account field.

You can change the field names to get the count of each occurrence like this:

index=main Account=* |eval Account1=mvindex(Account,0) |eval Account2=mvindex(Account,1) |stats count by Account1 Account2

The sum of each field will be the number of events.

[lukejadamec]

mvindex won't help with either one I'm afraid. It can help with events not getting counted twice, but cannot group events to be counted only once - that would be dedup or dc on a field in the 'event group' that is the same. It will not help with counters in a 'Message' field either - that would be rex to extract the part of the field of interest excluding the counter, or sed to rewrite or delete the counter.

[colbymahan]

Thanks for the replies. I will have to take some time to try understand how the suggestions apply. To clarify - I want to have certain Events not be counted as distinct due to a particular field value difference, like record number.
Secondly (and this is probably where the mvindex will help) I want to also do this at the Message level, so a particular part of the message (like an incremental counter) does not cause the Message to be distinct. I think there would be different methods for the 2 tasks.

[adylent]

You could use stream editor to alter the raw data, so the unique data is not present.

Example:
earliest=-1m index=my_index | rex mode=sed field=_raw "s/fieldname=\d{5}/#/"

[colbymahan]

This might be one way, but I have very little grasp on regex. I don't know how to format the field= part.

[linu1988]

could you post some events? or give us more clarity with your query?