topic Re: How do I formulate a regex to discard events when applying sourcetype? in Splunk Search

How do I formulate a regex to discard events when applying sourcetype?

andrewtrobec — Fri, 09 Feb 2018 11:04:06 GMT

Hello,
I'm currently creating a new sourcetype that has a TRANSFORMS-null setting with value discardit. Within my transforms.conf file I have created the appropriate stanza:

[discardit]
REGEX=(^|[\r\n])(.+,.+,.+,.+,"",.+)
DEST_KEY = queue
FORMAT = nullQueue

The regex that I've written is currently not working and I would like some help in formulating it.

The events are coming from csv files with the following header:

"Server","Job","Status","Start","End","Run Machine","Command"

My objective is to not index the events that have no value for End, so events that look like this:

"Server1","Job1","Running","2018-02-01 00:00:00","","Machine 1","Command 1"

The regex that I've written basically counts 4 commas and then checks to see whether the two characters after the fourth comma are "" (meaning that the value for Run is blank).

What am I missing?

Best regards,

Andrew

Re: How do I formulate a regex to discard events when applying sourcetype?

493669 — Fri, 09 Feb 2018 11:11:46 GMT

Hi @andrewtrobec,
you need to escape quotes by using back slashes
try this:

[discardit]
 REGEX=(^|[\r\n])(.+,.+,.+,.+,\"\",.+)
 DEST_KEY = queue
 FORMAT = nullQueue

Re: How do I formulate a regex to discard events when applying sourcetype?

FrankVl — Fri, 09 Feb 2018 11:17:52 GMT

Do you really need that complex bit with the \r\n?

Shouldn't this regex suffice (https://regex101.com/r/rbEHqz/1):

REGEX=^.+,.+,.+,.+,"",.+

Apart from that: what does your Splunk environment look like? Single instance, or distributed? If distributed, where did you deploy this config (should be on the first heavy instance, either HF or Indexer)?

Re: How do I formulate a regex to discard events when applying sourcetype?

andrewtrobec — Fri, 09 Feb 2018 13:27:35 GMT

Thanks for the suggestion. unfortunately adding the back slashes did not work. Do you have any other suggestions?

Re: How do I formulate a regex to discard events when applying sourcetype?

andrewtrobec — Fri, 09 Feb 2018 14:08:38 GMT

@FrankVl Thanks for the suggestion. This doesn't solve the problem, and I've no idea why. I will continue to make changes and test until something starts working.

My Splunk environment is a single instance. The config is in the $SPLUNK_HOME/etc/apps//local folder.

If you have any additional suggestions then please let me know!

Re: How do I formulate a regex to discard events when applying sourcetype?

micahkemp — Fri, 09 Feb 2018 14:19:26 GMT

Where is this props/transforms in place? On the indexer, heavy forwarder, universal forwarder, or search head?

Re: How do I formulate a regex to discard events when applying sourcetype?

493669 — Fri, 09 Feb 2018 14:29:59 GMT

Is your props.conf and transforms.conf files are in indexer/HWF?
I tried regex in splunk, it works as expected..

|makeresults|eval _raw="\"Server\",\"Job\",\"Status\",\"Start\",\"End\",\"\",\"Command\""|rex "(?<nulldata>(^|[\r\n])(.+,.+,.+,.+,\"\",.+))"

Re: How do I formulate a regex to discard events when applying sourcetype?

andrewtrobec — Fri, 09 Feb 2018 15:34:17 GMT

Yes, I am currently working on a single server instance. To test I am using the Add data wizard to see whether the lines disappear in the preview, and they do not. Am I correct to expect them to get removed there, or do I need to go ahead and index?

Re: How do I formulate a regex to discard events when applying sourcetype?

andrewtrobec — Fri, 09 Feb 2018 15:38:53 GMT

@micahkemp They are both in the $SPLUNK_HOME/etc/apps//local folder. It's a single server instance.