Getting Data In

Using only 1 event out of many duplicate events

abhaywdc
Loves-to-Learn

Greetings to you !!

I have a file in which I have a following content :

My city is very good

your city is also very good

but

but

but

but

Now, I want only three lines to be indexed in Splunk :

My city is very good

your city is also very good

but

Since "but" has appeared multiple times , so we want to use only 1 "but" out of many

I want to write props or any kind of configuration so that I can achieve this results.

Kindly help !!

0 Karma

KendallW
Communicator

Hi @abhaywdc there are a few ways to do this. Here's a way to do this using props.conf/transforms.conf:

props.conf:

 

...
TRANSFORMS-removeDupe=removeDupe

 

transforms.conf:

 

[removeDupe]
REGEX = (?s)(.*?)((but[\r\n]+)+)(.*)
FORMAT = $1$3$4
DEST_KEY = _raw

 

This transform tells Splunk to replace all the instances of "but" with the last instance, thereby de-duplicating them

Explanation of the regex from regexr:

KendallW_0-1719292658039.png

 

 

0 Karma
Get Updates on the Splunk Community!

Optimize Cloud Monitoring

  TECH TALKS Optimize Cloud Monitoring Tuesday, August 13, 2024  |  11:00AM–12:00PM PST   Register to ...

What's New in Splunk Cloud Platform 9.2.2403?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.2.2403! Analysts can ...

Stay Connected: Your Guide to July and August Tech Talks, Office Hours, and Webinars!

Dive into our sizzling summer lineup for July and August Community Office Hours and Tech Talks. Scroll down to ...