Getting Data In

remove specific data from csv data

chchanda
Loves-to-Learn

Hi There,

I have ingested the csv file via Splunk UF and I want to remove certain events that contains same field value, for example, field1 = xyz, abc, pqr,....     field2 = xyz

I want to send the data to null queue if field1 = xyz and field2 = xyz

This is my props.conf :

[<sourcetype>]
CHARSET = UTF-8
SHOULD_LINEMERGE = false
NO_BINARY_CHECK = true
LINE_BREAKER = ([\r\n]+)
NO_BINARY_CHECK = true
INDEXED_EXTRACTIONS = csv
KV_MODE = none
category = Structured
disabled = false
pulldown_type = true

Any help would be appreciated. Thanks 

Labels (2)
0 Karma

richgalloway
SplunkTrust
SplunkTrust

Removing events must be done by an indexer or heavy forwarder.  It can't be done by a UF.

You'll need a transform that tells Splunk which events to discard.  See https://docs.splunk.com/Documentation/Splunk/8.2.1/Forwarding/Routeandfilterdatad#Filter_event_data_... and https://community.splunk.com/t5/Getting-Data-In/Filtering-events-using-NullQueue/m-p/66392 for how to do that.

---
If this reply helps you, an upvote would be appreciated.
0 Karma

chchanda
Loves-to-Learn

, I am able to send the empty records to null queue but I also want to send few more to null queue(as cited in the query above). I have ingested the data using UF itself. 

Tags (2)
0 Karma

richgalloway
SplunkTrust
SplunkTrust

Data that is already ingested can't be removed until it expires.  You can hide events with the delete command, if you have that capability.

Since you already have the ability to send data to nullQueue, sending more there is just a matter of tweaking your regex or adding another transform.

---
If this reply helps you, an upvote would be appreciated.
0 Karma

chchanda
Loves-to-Learn

@richgalloway Here I cannot use the regex because the field1 value and field2 values are same. for example, 

field1field2
abc

abc

xyzabc
pqrabc
qweabc

 

I want to send the data where field1 = field2. can you please suggest?

Tags (2)
0 Karma

richgalloway
SplunkTrust
SplunkTrust

Depending on the props.conf settings, the transform likely is applied before field extractions so the regex should be based on the raw data rather than on individual fields.

---
If this reply helps you, an upvote would be appreciated.
0 Karma

chchanda
Loves-to-Learn

@richgalloway 

This is my props.conf :

[<sourcetype>]
CHARSET = UTF-8
SHOULD_LINEMERGE = false
NO_BINARY_CHECK = true
LINE_BREAKER = ([\r\n]+)
NO_BINARY_CHECK = true
INDEXED_EXTRACTIONS = csv
KV_MODE = none
category = Structured
disabled = false
pulldown_type = true
TRANSFORMS-set = eliminate-null_data
description=Comma-separated value format. Set header and other settings in "Delimited Settings"

transforms.conf:

[eliminate-null_data]
REGEX=<my regex>
DEST_KEY = queue
FORMAT = nullQueue

Note: the data flow can be stopped in order to send the desired data to null queue. 

Tags (2)
0 Karma

richgalloway
SplunkTrust
SplunkTrust

I'm not sure how well Splunk supports it, but you can do matching in regex by using back references.  This expression will match if field1 and field2 have the same value.

field1=(\w+), field2=\1

 

---
If this reply helps you, an upvote would be appreciated.
0 Karma
.conf21 Now Fully Virtual!
Register for FREE Today!

We've made .conf21 totally virtual and totally FREE! Our completely online experience will run from 10/19 through 10/20 with some additional events, too!