Splunk Search

How do you remove all duplicate events from a search?

strickland12345
Explorer

I have two indexes, A and B. Events are copied using the |collect command from Index A to index B. Later, I am trying to run a search for all results in index A that are not in index B. Something like:

index=A NOT index B

However this does not remove an event that is in both indexes. Essentially what I am trying is a |join type=left outer. However it seems that Splunk doesn't support that type of join. |Dedup seems to not recognize the events as duplicates either. I also tried using _cd as a unique identifier, however since that is tied to its location in the index, the two events have different _cd values preventing that from being used.

EDIT:

We currently are trying to allow users of our dashboard to "acknowledge" events. This process currently means filling in some input that sets tokens, which, on a drilldown action on a panel that has that event, runs a new search using |eval to append those tokens and |collect to move that event into index B.
The idea is that then we could make sure the our search for "to be acknowledged" will NOT include events that are in index B. Currently do to getting these issues I have been testing without the |eval bit, meaning that the two events are the exact same, the only difference is the index

0 Karma

strickland12345
Explorer

I have tried several queries, however the problem is I don't know of one that could do what I need as:

NOT does not remove both matching events
Dedup leaves atleast 1 event when removing duplicates
There is not join of type left outer

0 Karma

adonio
Ultra Champion

why are you collecting from index B to index a?
can you elaborate on your particular use case?
what is the problem you are trying to solve?

0 Karma

strickland12345
Explorer

My use case is this:

We currently are trying to allow users of our dashboard to "acknowledge" events. This process currently means filling in some input that sets tokens, which, on a drilldown action on a panel that has that event, runs a new search using |eval to append those tokens and |collect to move that event into index B.

The idea is that then we could make sure the our search for "to be acknowledged" will NOT include events that are in index B. Currently do to getting these issues I have been testing without the |eval bit, meaning that the two events are the exact same, the only difference is the index.

0 Karma
Got questions? Get answers!

Join the Splunk Community Slack to learn, troubleshoot, and make connections with fellow Splunk practitioners in real time!

Meet up IRL or virtually!

Join Splunk User Groups to connect and learn in-person by region or remotely by topic or industry.

Get Updates on the Splunk Community!

Why Splunk Customers Should Attend Cisco Live 2026 Las Vegas

Why Splunk Customers Should Attend Cisco Live 2026 Las Vegas     Cisco Live 2026 is almost here, and this ...

What Is the Name of the USB Key Inserted by Bob Smith? (BOTS Hint, Not the Answer)

Hello Splunkers,   So you searched, “what is the name of the usb key inserted by bob smith?”  Not gonna lie… ...

Automating Threat Operations and Threat Hunting with Recorded Future

    Automating Threat Operations and Threat Hunting with Recorded Future June 29, 2026 | Register   Is your ...