Dedup and multivalued fields

jamesdon — Wed, 09 Mar 2011 04:33:37 GMT

Hello,

I am extracting a few user names from a multiple line log, using MV_ADD=true in transforms.conf and KV_MODE = multi in transforms. The search results look great, and are what I am expecting; a single event per user found.

However, when I attempt to dedup the users, I run into problems. The user names are no longer individual events, but are consolidated into the original event. For instance, lets pretend that these are the three logs that I am searching:

Username: rabbit
Username: snail

Username: rabbit
Username: turtle

Username: rabbit

When I attempt to dedup the user name, I get:

rabbit, snail
rabbit, turtle
rabbit

I am expecting:

rabbit
snail
turtle

I thought that the problem would be resolved by using "multi: invokes multikv to expand a tabular event into multiple events" in props, but it does not.

Any ideas? I could always pre-process the data, but I was hoping to do it all in Splunk.

Thank you,

Jim

Re: Dedup and multivalued fields

Stephen_Sorkin — Wed, 09 Mar 2011 06:56:32 GMT

You have a couple choices here:

Use ... | mvexpand username to break each event into a distinct event per username, duplicating the message, followed by the dedup to find the most recent per username.
Use ... | stats count by username | fields username to just aggregate the list of username values.

The first is better if you want to see the original messages, but is less efficient than the second, which is better for just finding the user list.

topic Dedup and multivalued fields in Splunk Search

Dedup and multivalued fields

Re: Dedup and multivalued fields