topic Using only 1 event out of many duplicate events in Getting Data In

Using only 1 event out of many duplicate events

abhaywdc — Thu, 20 Jun 2024 19:40:20 GMT

Greetings to you !!

I have a file in which I have a following content :

My city is very good

your city is also very good

but

Now, I want only three lines to be indexed in Splunk :

My city is very good

your city is also very good

but

Since "but" has appeared multiple times , so we want to use only 1 "but" out of many

I want to write props or any kind of configuration so that I can achieve this results.

Kindly help !!

Re: Using only 1 event out of many duplicate events

KendallW — Tue, 25 Jun 2024 05:20:35 GMT

Hi @abhaywdc there are a few ways to do this. Here's a way to do this using props.conf/transforms.conf:

props.conf:

... TRANSFORMS-removeDupe=removeDupe

transforms.conf:

[removeDupe] REGEX = (?s)(.*?)((but[\r\n]+)+)(.*) FORMAT = $1$3$4 DEST_KEY = _raw

This transform tells Splunk to replace all the instances of "but" with the last instance, thereby de-duplicating them

Explanation of the regex from regexr: