Splunk Search

Remove duplicates in multiple columns (Column format - abc_000001 ) and sorting in same order with wild card?

Kukkadapu
Path Finder

Hi, I need some help to transform the below event? Thanks for your time.

2016-08-30 13:13:48,525 log_level='INFO' abc_000001="temp" abc_000002="temp1" abc_000003="temp2" abc_000004="temp2" abc_000005="temp2" abc_000006="temp3" ....... abc_000255="123" abc_000256="xyz"

| table abc_*

doesn't display all the fields and the order is missing. There are almost 300 fields. How to table all the field values using wild card?

How to create a new field - NEW_FIELD with the unique values of abc_* in the same order.

Expected Output :

NEW_FIELD

temp
temp1
temp2 (abc_000003,abc_000004 has the same value. So need to remove duplicates)
temp3
123
xyz

Thanks.

0 Karma
1 Solution

sundareshr
Legend

How about this

base search | extract pairdelim=" " kvdelim="=" | table abc_* | untable _hidden _current_field new_field | dedup new_field

View solution in original post

0 Karma

sundareshr
Legend

How about this

base search | extract pairdelim=" " kvdelim="=" | table abc_* | untable _hidden _current_field new_field | dedup new_field
0 Karma

Kukkadapu
Path Finder

Wow !! This works perfect 🙂 If you don't mind, can you explain the query?

0 Karma

sundareshr
Legend

I think you first problem was that the fields were not extracted properly, hence | table abc_* did not produce any results. The extract does just that. It extract kv pairs from _raw events.

https://docs.splunk.com/Documentation/Splunk/6.4.2/SearchReference/Extract

The untable command converts results from a tabular format to a format similar to stats output. I added a hidden field that, which has no values. But, but default all fields that begin with `are hidden by default. So naming a field starting with_is the same as adding| fields - hidden`

https://docs.splunk.com/Documentation/Splunk/6.4.3/SearchReference/Untable

Hope this helps. Feel free to ask questions if this doesn't clarify.

0 Karma

Kukkadapu
Path Finder

That makes sense. Thanks for your time sundareshr .

0 Karma
Get Updates on the Splunk Community!

Splunk Observability Cloud’s AI Assistant in Action Series: Analyzing and ...

This is the second post in our Splunk Observability Cloud’s AI Assistant in Action series, in which we look at ...

Elevate Your Organization with Splunk’s Next Platform Evolution

 Thursday, July 10, 2025  |  11AM PDT / 2PM EDT Whether you're managing complex deployments or looking to ...

Splunk Answers Content Calendar, June Edition

Get ready for this week’s post dedicated to Splunk Dashboards! We're celebrating the power of community by ...