topic Re: Unable to drop event in Getting Data In

Unable to drop event

alanhowlett — Tue, 29 Sep 2020 20:03:52 GMT

I'm trying to drop some failed messages from an imported txt file, but they continue to be in the sourcetype. The regex works using regexe101.com, and it looks ok in the splunk docs, but they wont be dropped.

Can somebody check my syntax please.

Below is copies of the transforms.conf and props.conf.
[splunk@linux1 local]$ cat props.conf
[sslah]
LINE_BREAKER = \d(4)-\d(2)-\d(2)\s\d(2):\d(2):\d(2)
TIME_FORMAT = %Y-%m-%d %H:%M:%S
TIME_PREFIX = ^
SHOULD_LINEMERGE = False
MAX_TIMESTAMP_LOOKAHEAD = 19
TRUNCATE = 2000
TRANSFORMS-null = setsslah

[splunk@linux1 local]$ cat transforms.conf
[setsslah]
REGEX = \d+.\d+.\d+.\d+\sFAILED
DEST_KEY = queue
FORMAT = nullQueue

Re: Unable to drop event

masonmorales — Thu, 14 Jun 2018 23:39:38 GMT

How is the data coming into Splunk? UF? HF? Standalone?

Re: Unable to drop event

cpetterborg — Fri, 15 Jun 2018 00:17:16 GMT

Check the configuration that it is using, in case there is another overriding configuration:

splunk btool transforms list --debug | less

Look for the setsslah stanzas in the output to see if it is getting the same thing that you think that it is.

Re: Unable to drop event

richgalloway — Tue, 29 Sep 2020 20:00:34 GMT

Your LINE_BREAKER attribute should use braces '{}' instead of parentheses '()'. Also, it must contain at least one capture group. Finally, everything that matches the LINE_BREAKER is discarded, which probably is not what you want with a timestamp.

Of course, all of this is pure speculation without seeing any sample data.

Re: Unable to drop event

amiftah — Fri, 15 Jun 2018 04:26:02 GMT

Which version of Splunk are you using?
Try with version 7.0.3

Re: Unable to drop event

alanhowlett — Fri, 15 Jun 2018 07:22:53 GMT

Good spot richgalloway, but changing to {} still doesn't work.

I thought it might be syntax so upgraded the software to Version 7.1.1

Still the same.

Re: Unable to drop event

alanhowlett — Fri, 15 Jun 2018 07:32:28 GMT

Also ran btool. No errors.

Re: Unable to drop event

FrankVl — Fri, 15 Jun 2018 07:33:09 GMT

You will want to escape the . if you want to actually match the . character: REGEX = \d+\.\d+\.\d+\.\d+\sFAILED

Although that doesn't really explain why it doesn't work. Because the . without escaping matches any character.

Re: Unable to drop event

FrankVl — Fri, 15 Jun 2018 07:36:44 GMT

btool doesn't show errors, it shows you the active configuration, to help detect possible conflicts / unexpected results.

Also: please explain the architecture and on what kind of Splunk instance you put this config.

Re: Unable to drop event

alanhowlett — Fri, 15 Jun 2018 08:02:54 GMT

Shouldnt it be \d+/.\d+/.\d+/.\d+\sFAILED

Re: Unable to drop event

FrankVl — Fri, 15 Jun 2018 08:07:52 GMT

No, you don't use forward slashes for escaping, you use \. And although it is probably a good idea to use \.instead of ., that should still work, since . in a regex will also match the literal . character.

Re: Unable to drop event

richgalloway — Fri, 15 Jun 2018 12:57:59 GMT

Please share some SAMPLE DATA. It's nearly impossible to debug regex strings without seeing what the regex is working with.

Re: Unable to drop event

alanhowlett — Tue, 29 Sep 2020 20:00:57 GMT

Still not working.

This is what I have now and I've double checked the regex.

root@e7c48e0e23c4:/opt/splunk/etc/apps/se2_tasks/local# cat props.conf
[sslbcoat1]
LINE_BREAKER = ^\d{4}-\d{2}-\d{2}\s\d{2}:\d{2}:\d{2}
MAX_TIMESTAMP_LOOKAHEAD = 19
SHOULD_LINEMERGE = true
TIME_FORMAT = %Y-%m-%d %H:%M:%S
TIME_PREFIX = ^
TRUNCATE = 2000
category = Custom
description = lab1
pulldown_type = true
TRANSFORMS-null = setnosslbcoat1

root@e7c48e0e23c4:/opt/splunk/etc/apps/se2_tasks/local# cat transforms.conf
[setnosslbcoat1]
REGEX = \d+.\d+.\d+.\d+\sFAILED
DEST_KEY = queue
FORMAT = nullQueue

Re: Unable to drop event

richgalloway — Sat, 16 Jun 2018 01:12:39 GMT

You still haven't shown any sample data.

The LINE_BREAKER attribute requires a capture group. Yours has none.