topic Re: how to drop events containing null values coming from a csv using transforms in Getting Data In

how to drop events containing null values coming from a csv using transforms

learnsplungeek — Fri, 21 Jun 2024 06:01:26 GMT

Hi
I am getting some events from a csv which contains the below format and would like to drop such events using transforms.

null,null,0,null,null,null,null,null,null, ---- to be dropped
null,null,0,null,null,null,null,null,null,  ---- to be dropped
null,null,0,null,null,null,null,null,null,  ---- to be dropped
null,null,0,null,null,null,null,null,null,  ---- to be dropped
null,null,0,null,null,null,null,null,null,  ---- to be dropped
52376,null,0,test,87387,2984,22,abc,99 ----- to be kept

Below is what i have done so far and is not working

Props.conf
[Reports5min]
TRANSFORMS-null = setnull

transforms.conf
[setnull]
REGEX = ^null,null\,0,null,null,null,null,null,null,$
DEST_KEY = queue
FORMAT = nullQueue

Re: how to drop events containing null values coming from a csv using transforms

learnsplungeek — Mon, 24 Jun 2024 05:43:42 GMT

Hi @woodcock @niketn

Please help me here

Re: how to drop events containing null values coming from a csv using transforms

glc_slash_it — Mon, 24 Jun 2024 07:28:25 GMT

Assuming the stanza [Reports5min] points to the right sourcetype, try placing a capture group in the regex property, like this:

REGEX = ^(null,null,0,null,null,null,null,null,null,)$

Also are you sure there is ni blacnk spaces at the end of each line?

The rest of the settinggs seems fine.

Re: how to drop events containing null values coming from a csv using transforms

learnsplungeek — Mon, 24 Jun 2024 08:41:26 GMT

Hi , Sure let me try this regex and see. Yes there are no blank spaces after each line containing null values

Re: how to drop events containing null values coming from a csv using transforms

PickleRick — Mon, 24 Jun 2024 13:21:04 GMT

1. Where are you putting those configs?

2. Do you use indexed extractions?

Re: how to drop events containing null values coming from a csv using transforms

learnsplungeek — Tue, 25 Jun 2024 06:51:27 GMT

I tried the regex and it did not work 😞

Re: how to drop events containing null values coming from a csv using transforms

learnsplungeek — Tue, 25 Jun 2024 06:52:56 GMT

Hi , I have placed both the transforms and props at indexer layer. We are getting the CSV data through UF's