Splunk Search

Remove duplicates in multiple columns (Column format - abc_000001 ) and sorting in same order with wild card?

Kukkadapu
Path Finder

Hi, I need some help to transform the below event? Thanks for your time.

2016-08-30 13:13:48,525 log_level='INFO' abc_000001="temp" abc_000002="temp1" abc_000003="temp2" abc_000004="temp2" abc_000005="temp2" abc_000006="temp3" ....... abc_000255="123" abc_000256="xyz"

| table abc_*

doesn't display all the fields and the order is missing. There are almost 300 fields. How to table all the field values using wild card?

How to create a new field - NEW_FIELD with the unique values of abc_* in the same order.

Expected Output :

NEW_FIELD

temp
temp1
temp2 (abc_000003,abc_000004 has the same value. So need to remove duplicates)
temp3
123
xyz

Thanks.

0 Karma
1 Solution

sundareshr
Legend

How about this

base search | extract pairdelim=" " kvdelim="=" | table abc_* | untable _hidden _current_field new_field | dedup new_field

View solution in original post

0 Karma

sundareshr
Legend

How about this

base search | extract pairdelim=" " kvdelim="=" | table abc_* | untable _hidden _current_field new_field | dedup new_field
0 Karma

Kukkadapu
Path Finder

Wow !! This works perfect 🙂 If you don't mind, can you explain the query?

0 Karma

sundareshr
Legend

I think you first problem was that the fields were not extracted properly, hence | table abc_* did not produce any results. The extract does just that. It extract kv pairs from _raw events.

https://docs.splunk.com/Documentation/Splunk/6.4.2/SearchReference/Extract

The untable command converts results from a tabular format to a format similar to stats output. I added a hidden field that, which has no values. But, but default all fields that begin with `are hidden by default. So naming a field starting with_is the same as adding| fields - hidden`

https://docs.splunk.com/Documentation/Splunk/6.4.3/SearchReference/Untable

Hope this helps. Feel free to ask questions if this doesn't clarify.

0 Karma

Kukkadapu
Path Finder

That makes sense. Thanks for your time sundareshr .

0 Karma
Get Updates on the Splunk Community!

Demo Day: Strengthen Your SOC with Splunk Enterprise Security 8.1

Today’s threat landscape is more complex than ever. Security operation centers (SOCs) are overwhelmed with ...

Dashboards: Hiding charts while search is being executed and other uses for tokens

There are a couple of features of SimpleXML / Classic dashboards that can be used to enhance the user ...

Splunk Observability Cloud's AI Assistant in Action Series: Explaining Metrics and ...

This is the fourth post in the Splunk Observability Cloud’s AI Assistant in Action series that digs into how ...