Splunk Search

Remove duplicates in multiple columns (Column format - abc_000001 ) and sorting in same order with wild card?

Kukkadapu
Path Finder

Hi, I need some help to transform the below event? Thanks for your time.

2016-08-30 13:13:48,525 log_level='INFO' abc_000001="temp" abc_000002="temp1" abc_000003="temp2" abc_000004="temp2" abc_000005="temp2" abc_000006="temp3" ....... abc_000255="123" abc_000256="xyz"

| table abc_*

doesn't display all the fields and the order is missing. There are almost 300 fields. How to table all the field values using wild card?

How to create a new field - NEW_FIELD with the unique values of abc_* in the same order.

Expected Output :

NEW_FIELD

temp
temp1
temp2 (abc_000003,abc_000004 has the same value. So need to remove duplicates)
temp3
123
xyz

Thanks.

0 Karma
1 Solution

sundareshr
Legend

How about this

base search | extract pairdelim=" " kvdelim="=" | table abc_* | untable _hidden _current_field new_field | dedup new_field

View solution in original post

0 Karma

sundareshr
Legend

How about this

base search | extract pairdelim=" " kvdelim="=" | table abc_* | untable _hidden _current_field new_field | dedup new_field
0 Karma

Kukkadapu
Path Finder

Wow !! This works perfect 🙂 If you don't mind, can you explain the query?

0 Karma

sundareshr
Legend

I think you first problem was that the fields were not extracted properly, hence | table abc_* did not produce any results. The extract does just that. It extract kv pairs from _raw events.

https://docs.splunk.com/Documentation/Splunk/6.4.2/SearchReference/Extract

The untable command converts results from a tabular format to a format similar to stats output. I added a hidden field that, which has no values. But, but default all fields that begin with `are hidden by default. So naming a field starting with_is the same as adding| fields - hidden`

https://docs.splunk.com/Documentation/Splunk/6.4.3/SearchReference/Untable

Hope this helps. Feel free to ask questions if this doesn't clarify.

0 Karma

Kukkadapu
Path Finder

That makes sense. Thanks for your time sundareshr .

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...