How do I formulate a regex to discard events when applying sourcetype?

[andrewtrobec]

Hello,
I'm currently creating a new sourcetype that has a TRANSFORMS-null setting with value discardit. Within my transforms.conf file I have created the appropriate stanza:

[discardit]
REGEX=(^|[\r\n])(.+,.+,.+,.+,"",.+)
DEST_KEY = queue
FORMAT = nullQueue

The regex that I've written is currently not working and I would like some help in formulating it.

The events are coming from csv files with the following header:

"Server","Job","Status","Start","End","Run Machine","Command"

My objective is to not index the events that have no value for End, so events that look like this:

"Server1","Job1","Running","2018-02-01 00:00:00","","Machine 1","Command 1"

The regex that I've written basically counts 4 commas and then checks to see whether the two characters after the fourth comma are "" (meaning that the value for Run is blank).

What am I missing?

Best regards,

Andrew

[micahkemp]

Where is this props/transforms in place? On the indexer, heavy forwarder, universal forwarder, or search head?

[andrewtrobec]

@micahkemp They are both in the $SPLUNK_HOME/etc/apps//local folder. It's a single server instance.

[FrankVl]

Do you really need that complex bit with the \r\n?

Shouldn't this regex suffice (https://regex101.com/r/rbEHqz/1):

REGEX=^.+,.+,.+,.+,"",.+

Apart from that: what does your Splunk environment look like? Single instance, or distributed? If distributed, where did you deploy this config (should be on the first heavy instance, either HF or Indexer)?

[andrewtrobec]

@FrankVl Thanks for the suggestion. This doesn't solve the problem, and I've no idea why. I will continue to make changes and test until something starts working.

My Splunk environment is a single instance. The config is in the $SPLUNK_HOME/etc/apps//local folder.

If you have any additional suggestions then please let me know!

[493669]

Hi @andrewtrobec,
you need to escape quotes by using back slashes
try this:

[discardit]
 REGEX=(^|[\r\n])(.+,.+,.+,.+,\"\",.+)
 DEST_KEY = queue
 FORMAT = nullQueue

[andrewtrobec]

Thanks for the suggestion. unfortunately adding the back slashes did not work. Do you have any other suggestions?

[493669]

Is your props.conf and transforms.conf files are in indexer/HWF?
I tried regex in splunk, it works as expected..

|makeresults|eval _raw="\"Server\",\"Job\",\"Status\",\"Start\",\"End\",\"\",\"Command\""|rex "(?<nulldata>(^|[\r\n])(.+,.+,.+,.+,\"\",.+))"

[andrewtrobec]

Yes, I am currently working on a single server instance. To test I am using the Add data wizard to see whether the lines disappear in the preview, and they do not. Am I correct to expect them to get removed there, or do I need to go ahead and index?