topic Re: remove specific data from csv data in Getting Data In

remove specific data from csv data

chchanda — Tue, 20 Jul 2021 13:58:08 GMT

Hi There,

I have ingested the csv file via Splunk UF and I want to remove certain events that contains same field value, for example, field1 = xyz, abc, pqr,.... field2 = xyz

I want to send the data to null queue if field1 = xyz and field2 = xyz

This is my props.conf :

Any help would be appreciated. Thanks

Re: remove specific data from csv data

richgalloway — Fri, 16 Jul 2021 20:43:50 GMT

Removing events must be done by an indexer or heavy forwarder. It can't be done by a UF.

You'll need a transform that tells Splunk which events to discard. See https://docs.splunk.com/Documentation/Splunk/8.2.1/Forwarding/Routeandfilterdatad#Filter_event_data_and_send_to_queues and https://community.splunk.com/t5/Getting-Data-In/Filtering-events-using-NullQueue/m-p/66392 for how to do that.

Re: remove specific data from csv data

chchanda — Tue, 20 Jul 2021 14:47:14 GMT

richgalloway, I am able to send the empty records to null queue but I also want to send few more to null queue(as cited in the query above). I have ingested the data using UF itself.

Re: remove specific data from csv data

richgalloway — Tue, 20 Jul 2021 12:11:49 GMT

Data that is already ingested can't be removed until it expires. You can hide events with the delete command, if you have that capability.

Since you already have the ability to send data to nullQueue, sending more there is just a matter of tweaking your regex or adding another transform.

Re: remove specific data from csv data

chchanda — Tue, 20 Jul 2021 12:37:42 GMT

@richgalloway Here I cannot use the regex because the field1 value and field2 values are same. for example,

field1	field2
abc	abc
xyz	abc
pqr	abc
qwe	abc

I want to send the data where field1 = field2. can you please suggest?

Re: remove specific data from csv data

richgalloway — Tue, 20 Jul 2021 13:56:17 GMT

Depending on the props.conf settings, the transform likely is applied before field extractions so the regex should be based on the raw data rather than on individual fields.

Re: remove specific data from csv data

chchanda — Tue, 20 Jul 2021 14:01:17 GMT

@richgalloway

This is my props.conf :

[<sourcetype>]
CHARSET = UTF-8
SHOULD_LINEMERGE = false
NO_BINARY_CHECK = true
LINE_BREAKER = ([\r\n]+)
NO_BINARY_CHECK = true
INDEXED_EXTRACTIONS = csv
KV_MODE = none
category = Structured
disabled = false
pulldown_type = true
TRANSFORMS-set = eliminate-null_data
description=Comma-separated value format. Set header and other settings in "Delimited Settings"

transforms.conf:

[eliminate-null_data]
REGEX=<my regex>
DEST_KEY = queue
FORMAT = nullQueue

Note: the data flow can be stopped in order to send the desired data to null queue.

Re: remove specific data from csv data

richgalloway — Tue, 20 Jul 2021 18:16:20 GMT

I'm not sure how well Splunk supports it, but you can do matching in regex by using back references. This expression will match if field1 and field2 have the same value.

field1=(\w+), field2=\1