Using only 1 event out of many duplicate events

[abhaywdc]

Greetings to you !!

I have a file in which I have a following content :

My city is very good

your city is also very good

but

but

but

but

Now, I want only three lines to be indexed in Splunk :

My city is very good

your city is also very good

but

Since "but" has appeared multiple times , so we want to use only 1 "but" out of many

I want to write props or any kind of configuration so that I can achieve this results.

Kindly help !!

[KendallW]

Hi @abhaywdc there are a few ways to do this. Here's a way to do this using props.conf/transforms.conf:

props.conf:

...
TRANSFORMS-removeDupe=removeDupe

transforms.conf:

[removeDupe]
REGEX = (?s)(.*?)((but[\r\n]+)+)(.*)
FORMAT = $1$3$4
DEST_KEY = _raw

This transform tells Splunk to replace all the instances of "but" with the last instance, thereby de-duplicating them

Explanation of the regex from regexr: